<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yu Wei Chung on Medium]]></title>
        <description><![CDATA[Stories by Yu Wei Chung on Medium]]></description>
        <link>https://medium.com/@ychung38?source=rss-a81b27e7e0a3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*c4Gsz2ok14ezhMnK</url>
            <title>Stories by Yu Wei Chung on Medium</title>
            <link>https://medium.com/@ychung38?source=rss-a81b27e7e0a3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 23:33:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ychung38/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Sweetviz — a library can let you analyze better and quicker]]></title>
            <link>https://ychung38.medium.com/sweetviz-a-library-can-let-you-analyze-better-and-quicker-7ea10bb935bc?source=rss-a81b27e7e0a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ea10bb935bc</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Yu Wei Chung]]></dc:creator>
            <pubDate>Sun, 01 Nov 2020 14:06:20 GMT</pubDate>
            <atom:updated>2020-11-01T14:06:20.278Z</atom:updated>
            <content:encoded><![CDATA[<h3>Sweetviz — a library can let you analyze better and quicker</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*k38CwNBAz7vrJ4ru8Mns5w.png" /><figcaption><a href="https://unsplash.com/photos/AT77Q0Njnt0">Unsplash</a></figcaption></figure><p>How to visualize a dataset in a general way for the first time? Before I learn this library, I wrote almost the same code to try to find a good plot to let me find some good thoughts in the data. Using my analysis ability help me discover the nature of findings, but writing the indifferent code is not.</p><p>That’s why <a href="https://github.com/fbdesignpro/sweetviz">Sweetviz</a> is really important for data analyst, data scientist and every people that need to analyze data or find something interesting in the data.</p><p>The process in this article:</p><ol><li><strong>Introduction</strong></li><li><strong>Demo</strong></li><li><strong>Conclusion</strong></li></ol><h3>1. Introduction</h3><p>This is the summary from <a href="https://github.com/fbdesignpro/sweetviz">Sweetviz</a></p><blockquote>Sweetviz is an open source Python library that generates beautiful, high-density visualizations to kickstart EDA (Exploratory Data Analysis) with a single line of code. Output is a fully self-contained HTML application.</blockquote><p>We can understand this tool can help us plot a lot of figures from each variable and the target from the first time we get the dataset. After using the Sweetviz, you can analyze each figure, find the most influencing variable and start to dig deeper.</p><h4>Sweetviz can help you</h4><p>(1) do the basic EDA (Exploratory Data Analysis) from the variable and targets and calculate statistical results quickly.</p><p>(2) Compare the distribution of the features in two different datasets.</p><p>(3) Compare the distribution of the features in two subsets from a dataset</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*M9SBSwKSJLP9dLZRS4bLbA.png" /><figcaption><a href="https://github.com/fbdesignpro/sweetviz">The sample from Sweetviz Github</a></figcaption></figure><h3>2. Demo</h3><p>You should install Sweetviz first!</p><pre>pip install sweetviz</pre><p>There is a thing I need to remind you of. The volume of a dataset and the number of variables can have big or small. It the total count is too big, you can cancel the correlation computation. Therefore, if you don’t want the correlation result, you can add a parameter when you analyze or compare:</p><pre>pairwise_analysis=&#39;off&#39;</pre><p>The dataset I use is <a href="https://www.kaggle.com/c/titanic">Titanic</a> from Kaggle. The reason that I choose this dataset is that it is the easiest dataset in the data science world. You can spend less time to familiarize this dataset because you are familiar with this.</p><p>Before you use Sweetviz, you should read the data</p><pre>train = pd.read_csv(&#39;./titanic/train.csv&#39;, index_col=0)<br>test = pd.read_csv(&#39;./titanic/test.csv&#39;, index_col=0)</pre><pre># You also need to import the library<br>import sweetviz as sv</pre><h4>1. Basic report</h4><p>This means you don’t need to change or add a lot of parameters. You can just use the default parameter.</p><pre>report_train = sv.analyze([train, &#39;train&#39;]) # &#39;train&#39;是指會給這個資料集命名為train<br>report_train.show_html(filepath=&#39;Basic_train_report.html&#39;) # 儲存為html的格式</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sUteZRs0GXDwNbFT5eL9bg.png" /></figure><h4>2. Report that is added target variable (Y variable)</h4><p>If you already use Sweetviz, you can see that Sweetviz will determine the variable type which maybe is numerical, text, or categorical. The type that is decided for you might is wrong, so there is a way to help you change the default.</p><pre>feature_config = sv.FeatureConfig(skip=&#39;Name&#39;,  # the variable you want to ignore<br>                                  force_cat=[&#39;Pclass&#39;, &#39;Sex&#39;, &#39;SibSp&#39;, &#39;Parch&#39;, &#39;Ticket&#39;, &#39;Cabin&#39;], # Categorical <br>                                  force_num=[&#39;Age&#39;, &#39;Fare&#39;], # Numerical<br>                                  force_text=None) # Text</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m6lvXrZxWPQE4EmCBDQX0g.png" /></figure><h4>3. Report comparing train and test</h4><p>We usually need to compare the train and test datasets to make sure the dataset doesn’t have an overfitting or underfitting situation.</p><pre>compare_report = sv.compare([train, &#39;Training Data&#39;], # compare<br>                            [test, &#39;Test Data&#39;],<br>                            &#39;Survived&#39;,<br>                            feat_cfg=feature_config)<br><br>compare_report.show_html(filepath=&#39;Compare_train_test_report.html&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zWE2Wwfn0B2Rwc5XQJCKQg.png" /></figure><h4>4. Report comparing Male and Female subsets</h4><p>We also have a chance to compare the binary situation in a variable in the same dataset. By comparing two subsets, we can check other variables whether have different distributions or not.</p><pre>compare_subsets_report = sv.compare_intra(train,<br>                                          train[&#39;Sex&#39;]==&#39;male&#39;, # Give condition<br>                                          [&#39;Male&#39;, &#39;Female&#39;], # name for each subset<br>                                          target_feat=&#39;Survived&#39;,<br>                                          feat_cfg=feature_config)<br><br>compare_subsets_report.show_html(filepath=&#39;Compare_male_female_report.html&#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oW-6MrfBwogGB6bbOg0CgA.png" /></figure><h3>3. Conclusion</h3><p>By this article, I think you can understand how to use this good library. However, the most important part of the analysis that you need to remember is not the report you create. It’s the insights you get from the report.</p><p>Thank you for sticking to the end. I love sharing my data science knowledge and hope you learn a lot from me. Strive for greatness!</p><p>If you have any questions, send the questions to my email jacky308082@gmail.com.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ea10bb935bc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use Python+Metaflow to build a perfect data science project]]></title>
            <link>https://ychung38.medium.com/use-python-metaflow-to-build-a-perfect-data-science-project-9458ef1b5a30?source=rss-a81b27e7e0a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/9458ef1b5a30</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Yu Wei Chung]]></dc:creator>
            <pubDate>Mon, 26 Oct 2020 13:13:27 GMT</pubDate>
            <atom:updated>2020-10-26T13:13:27.034Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*Hj-sDnsmGzWMrn-PULRuvQ.png" /><figcaption><a href="https://docs.metaflow.org/">Metaflow</a></figcaption></figure><p>Hello everyone, this time I will introduce a very powerful open-source tool in Python. <a href="https://docs.metaflow.org/">Metaflow</a> is created by Netflix. It can help you build and manage your own or team’s data science project in a more quickly way.</p><p>If you or your team has more than one project, every project needs to be dealt with many things back and forth. At the same time, you also need to optimize the model. For example, one project should contain these processes:</p><p>(1) Define a specific business problem</p><p>(2) Try to analyze and propose an executable plan based on the business problem</p><p>(3) Use machine learning algorithms to solve the problem</p><p>(4) Deploy models to the server</p><p>(5) Models is used to predict new data and be fit by new data to keep the newest status</p><p>(6) Get new insights from the new result</p><p>(7) Bring the new insights and back to (1) step</p><h3>1. Introduction to Metaflow</h3><p>There is an introduction about Metaflow from Netflix</p><blockquote><em>Metaflow is a human-friendly Python library that helps scientists and engineers build and manage real-life data science projects. Metaflow was originally developed at Netflix to boost productivity of data scientists who work on a wide variety of projects from classical statistics to state-of-the-art deep learning.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wAMbgBBYUJOcxuJxqninUA.png" /><figcaption>Data Scientists hope &amp; Architecturers want</figcaption></figure><p>This tool that is not like machine learning algorithms can boost your project performance directly, but it can increase your productivity. When you deal with a project, you need to take a lot of time to figure out the process and make sure the flow in the code. Therefore, Metaflow can help us to accomplish these things and allow us to focus the analysis and modeling. In fact, Metaflow also increase the perfomance of model indirectly.</p><h3>2. Dataset</h3><p>The dataset I used is <a href="https://archive.ics.uci.edu/ml/datasets/Bank+Marketing">Bank Marketing Dataset</a> in the UCI.</p><h3>3. Metaflow Example</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*uBEXcG7-Krhrs5QNcQ6f0g.png" /><figcaption>Metaflow Launch</figcaption></figure><p>This picture is showed when the Metaflow launched. You can use the flow chart in the left side to see the code’s execution process. Right side is the process’s code. The process contains:</p><p>(1) Start</p><p>(2) A, B step</p><p>(3) after finish, two processes merge to ‘join’</p><p>(4) End in the ‘End’</p><p>After writing the code, you can open your cmd and key:</p><pre>python myscript.py run</pre><p>Then you will get the result.</p><p>In actual fact, the real Metaflow will not produce the left side picture, but you can run the following code to review your process.</p><pre>python myscript.py show</pre><h3>5. Use metaflow to build project</h3><p>First of all, the most important of data analysis is data. We need to specify the data storage address. I save data in ‘bank-additional’ file.</p><pre>def script_path(filename):<br>     import os<br>     filepath = os.path.join(os.path.dirname(__file__))<br>     return os.path.join(filepath, &#39;bank-additional&#39;, filename)</pre><p>You need to wrap your project to “class” type based on each step. You can write your code very detailed, because if the code has error, you can find out the step immediately.</p><pre>from metaflow import FlowSpec, IncludeFile, step<br>import pandas as pd</pre><p>The reason that I import pandas at the firt is I need to use pandas’s function in almost every step to do necessary data cleaning. Otherwise, I have to import pandas every step I used. It will take too long to do.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/569a4eab1fa2acfb03bee03967aa62be/href">https://medium.com/media/569a4eab1fa2acfb03bee03967aa62be/href</a></iframe><p>All explaination about Metaflow is all in the above script. Remeber you need add the code</p><pre>if __name__ == &#39;__main__&#39;:<br>    BankMarketing()</pre><p>to make you wil get the result after you run this .py file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/238/1*p4RRGvtCbyYA1UjlzBmfaw.png" /></figure><p>Everying is looking fine. And then we can run whole step.</p><pre>python Project_metaflow.py run</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sj32SM7cwXsZABO0TKLa1g.png" /></figure><p>Looking ver food! The end is here. When you face any data science project that can be optimized, you can use the strong tool to boost your productivity. It can significantly save your time to wrtie structured code and review code and let your more focus on analysis and domain knowledge.</p><p>Thank you for sticking to the end. I love sharing my data science knowledge and hope you learn a lot from me. Strive for greatness!</p><p>If you have any questions, send the questions to my email jacky308082@gmail.com.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9458ef1b5a30" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use Solver (Excel) in Python]]></title>
            <link>https://ychung38.medium.com/how-to-use-solver-excel-in-python-458336408c7f?source=rss-a81b27e7e0a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/458336408c7f</guid>
            <category><![CDATA[problem-solving]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[excel]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Yu Wei Chung]]></dc:creator>
            <pubDate>Fri, 09 Oct 2020 04:10:36 GMT</pubDate>
            <atom:updated>2020-10-09T04:10:36.675Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7RBLSfColPBx02IMOSqMPA.jpeg" /><figcaption><a href="https://medium.com/aubergine-solutions/working-with-excel-sheets-in-python-using-openpyxl-4f9fd32de87f">Source</a></figcaption></figure><p>My <a href="https://www.linkedin.com/in/yu-wei-chung/">LinkedIn</a></p><p>Many times when you having to solve an optimization problem (Linear Programming), you may think of the Solver in Excel at the first time. Today, I want to introduce the library in Python that has as powerful as the Solver. The pulp is a good one you can use if you are facing an optimization problem.</p><p>And you may not know what is the Solver. This article I will begin with the Solver in Excel. After that, I will change the method from Excel to Python to solve the same problem.</p><h3>Steps</h3><ol><li>Problem</li><li>Introduce Solver in Excel</li><li>Solve the same problem in Python</li><li>Conclusion</li></ol><h3>Problem</h3><p>There is a T-shirt manufacturer. They produce two kinds of T-shirts. One is a T-shirt that is made of silk. The other is one that is made in cotton. Silk T-shirt per item sells $45 and costs $30. The cotton T-shirt per item sells $20 and costs $10. To meet customer&#39;s demand, at least 2000 units of silk T-shirt and at least 8000 units of cotton T-shirt need to be produced and sold. The manufacturer can produce 8000 units of two kinds of T-shirts at most. What is the maximum of revenue that manufacture can make?</p><h3>Solver In Excel</h3><p>If you want to deal with the optimum problem, you need to write down your variable, objective, and constraints. This is what I learn for the first time. Furthermore, let me write down the problem in Excel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*KnK4ZUCDPNp7ndLxI5qC1Q.png" /></figure><p>You can get the same cell if you read the question. And then, I set the variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*Kzk9VPrAaSpQDDkW9s_dCA.png" /></figure><p>Next, it’s an objective function to get the maximum value. max_revenue can be computed by Price(Silk)*x1 + Price(Cotton)*x2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*d4BzouMzS0999-k89C5A8Q.png" /></figure><p>After finishing these settings, you can start to use the Solver to solve this optimization problem. If you don’t find the Solver, you can click the Add-in to add the Solver.</p><p>You should set the variable and objective function before you run the Solver and consider the constraint that the problem gives.</p><p>This is the screenshot after add each condition in Solver.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*TjO-B8Ps6vwwlFUGS3sPcw.png" /></figure><p>This is the Chinese version. You can depend on the Solver and translate easily. I set the objective, variable, and constraints, and then I just click the Solve. Excel will solve this problem for me.</p><p>That’s the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*mrpp8p8kjt_cOsGV4PEXog.png" /></figure><h3>Solve the same problem in Python</h3><p>In the previous part, we talked about how to use Solver to solve an optimal question. Therefore, we also can do the same thing in Python using Pulp library.</p><ol><li>Create a Problem DataFrame</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba3a900001b380792fbf7db99ce16f63/href">https://medium.com/media/ba3a900001b380792fbf7db99ce16f63/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*uOZIrjZTbyNxUHUuBYK3NQ.png" /></figure><p>2. Set the problem</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5dcf96a04a4f03f22cea175c3f550146/href">https://medium.com/media/5dcf96a04a4f03f22cea175c3f550146/href</a></iframe><p>That’s it. We set everything about the problem such as the objective, variables, constraints. We check the ‘prob’ again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*bJf4cJRC6sBCndFeJMJjAw.png" /></figure><pre># Run this. Like click the solve to let Solver run.<br>prob.solve()</pre><p>After we run the solving code, we must be curious about what is the answer to the objective and the value of variables.</p><pre># Answer<br>value(prob.objective) # 9000</pre><pre># Variables&#39; values<br>print(&#39;The optimal answer\n&#39;+&#39;-&#39;*70)<br>for v in prob.variables():<br>    if v.varValue &gt; 0:<br>       print(v.name, &#39;=&#39;, v.varValue)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PkCDU1BJvuAbUcox6bIk0A.png" /></figure><p>Look! It is as same as the answer we get in Solver. And it is more easier to do it.</p><h3>Conclusion</h3><p>You might want to ask me which is better to solve the optimization problem. For me, I will choose Python to solve, because I love Python’s versatility. I can do machine learning, execute feature engineering, and solve the optimization problems at the same time. That’s why I choose Python instead of excel.</p><p>You can ask me everything. Send the question to my email jacky308082@gmail.com.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=458336408c7f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intro to MLflow in Python]]></title>
            <link>https://ychung38.medium.com/intro-to-mlflow-in-python-db53ba8a614a?source=rss-a81b27e7e0a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/db53ba8a614a</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Yu Wei Chung]]></dc:creator>
            <pubDate>Sat, 26 Sep 2020 14:32:00 GMT</pubDate>
            <atom:updated>2020-09-26T14:32:00.112Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*ndj50szEmTnFqWEnTdEkaw.png" /><figcaption>source: <a href="https://devclass.com/2019/07/24/mlflow-highlights-logging-and-tracking-in-v1-1/">https://devclass.com/2019/07/24/mlflow-highlights-logging-and-tracking-in-v1-1/</a></figcaption></figure><p>This time I want to talk about MLflow. MLflow is an open-source platform to manage the machine learning lifecycle, including experimentation, reproducibility, and deployment. You can check the <a href="https://www.mlflow.org/">website</a> for more information.</p><p>MLflow can be separated into tracking, projects, and models.</p><p>Tracking: Tracking allows you to create an extensive logging framework around your model. You can define custom metrics so that after a run you can compare the output to previous runs.</p><p>Projects: This feature allows you to create a pipeline if you so desire. This feature uses its own template to define how you want to run the model on a cloud environment.</p><p>Models: This feature is a standard format for packaging machine learning models that can be used in a variety of downstream tools.</p><p>This article will use the Tracking method to express the machine learning result on your own server.</p><p>I would the steps to build up an MLflow online model.</p><h4>1. Dataset</h4><p>I would use the Iris to the dataset in this article because it is extremely to explain and build a machine learning model. In the article, I would not focus on how the accuracy or any metric is. I do not pursue a better machine learning model too. Just show how to use MLflow to see the result and you can make it yourself after reading this article.</p><h4>2. Start with the import</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aec16d01a2deb11119f6ab4392246f3d/href">https://medium.com/media/aec16d01a2deb11119f6ab4392246f3d/href</a></iframe><h4>3. Create the evaluated function</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3d89c400342d1770cc3b8b59c4b601fb/href">https://medium.com/media/3d89c400342d1770cc3b8b59c4b601fb/href</a></iframe><h4>4. Data preprocessing</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/27404dc8b0a25c391a587e5627e97ec4/href">https://medium.com/media/27404dc8b0a25c391a587e5627e97ec4/href</a></iframe><h4>5. Different hyperparameters</h4><p>I would use the different hyperparameters to express different results.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/79d006fae3299ab7c330bfc7212d9ffb/href">https://medium.com/media/79d006fae3299ab7c330bfc7212d9ffb/href</a></iframe><h4>6. Using the MLflow to see the result</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/352f17e0a15d92132b3ca1062542a801/href">https://medium.com/media/352f17e0a15d92132b3ca1062542a801/href</a></iframe><pre># Next step you can write the file name to run in the cmd.<br>python name.py<br>mlflow ui# Search <a href="http://127.0.0.1:5000/#/">http://127.0.0.1:5000/#/</a></pre><p>You will get a website like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8EdVnhKeSVlHsHa5gjUU0Q.png" /><figcaption>MLflow</figcaption></figure><p>It is easy to build this. You can try some datasets you have right now to change to MLflow. You can check the result on the web page and see the hyperparameters clearly.</p><p>The article is very easy to follow. I also welcome any feedback. You can email me jacky308082@gmail.com. Let’s strive for greatness!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db53ba8a614a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Analytics Customer Revenue Prediction -Kaggle challenge (Part Three)]]></title>
            <link>https://ychung38.medium.com/google-analytics-customer-revenue-prediction-kaggle-challenge-part-three-24ff1cece9b0?source=rss-a81b27e7e0a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/24ff1cece9b0</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[problem-solving]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Yu Wei Chung]]></dc:creator>
            <pubDate>Tue, 22 Sep 2020 16:14:27 GMT</pubDate>
            <atom:updated>2020-09-22T16:14:27.061Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ukPRAoaw8HYnFk21yHZQYg.jpeg" /><figcaption><a href="https://blog.xuite.net/cationi/blog/79674328-%5B%E5%85%A7%E6%B9%96%5D+%E6%B9%96%E5%B1%B1%E5%84%AA%E7%BE%8E%E7%9A%84%E4%B8%80%E5%8D%80">Source</a></figcaption></figure><p>This is the third post in this series of articles. In this article, I would introduce how to implement the cross-validation and lightGBM algorithm. For me, it is not an easy way to explain the algorithm thoroughly, but I would try. I make sure that you would know how the algorithm works and apply it to your own data science project.</p><p>In the last article, we discuss the <strong>exploratory data analysis</strong> and <strong>feature engineering</strong>. The first one gives the second important insight. In every data science project, we should spend the most time to deal with these two sections, but it’s worth it.</p><h3>Model optimization</h3><p>Before I start to explain this part, you would ask the model value if we did not make a feature engineering. In this competition. If you did the right feature engineering as I did in the last article, the model after implementing feature engineering would just a little better than the raw model. But the truth is the model is improved indeed, and you could make a progress in the rank in the competition. Therefore, I would show two versions of the model. One is the raw model, and the other is the revised model. Let’s start!</p><h4>Base Model</h4><p>You should forget the last article&#39;s content. Do you still remember it? Okay, I think your mind is clear. To find the model predicted ability, I used the KFold to help me split the train datasets into 5 pieces. When I want to use cross-validation to evaluate the data, I could use four of the pieces to train data and remained piece to test data and implement five different train data in training.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1032554be9c1341c868e191ece48a5f1/href">https://medium.com/media/1032554be9c1341c868e191ece48a5f1/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*cARyLc8w7AZT7PqFiCwEmQ.png" /></figure><p>This is a little bit of annoy, but you could use the same call in every machine learning project. I recommend you could deeply learn it in just one time, and then you are not worried about anything model evaluation. Therefore, if you follow my code to do it, you could do other things you do not finish because you have to wait for the result. And it is the result.</p><h4>Sophisticated Model</h4><p>Here I would just post the result because the code is as same as the base model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*EZg1Q9ylrINPOJDv93UE5g.png" /></figure><p>You could see each rmse. It would slightly better than before, but for the competition, a better score means better.</p><p>I would also show the feature importance in base train and sophisticated model. Let you see what the features are important for the model.</p><ol><li>Base Model</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lTivmvXvzIkZAfwSzHXxvA.png" /></figure><p>2. Sophisticated Model</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qBo9RNq-Xiowfz5gCtSRSg.png" /></figure><p>And then, after we finished the model, we should show the time we spent in this code and the types of rmse. In the validation evaluation, we see that we added the value in each iteration. ‘valid_score’ should be divided by spliting number. Therefore, we would see two results. One is ‘valid_score’ / n_splits. It represents the rmse of this session. The other one is the result that the competition wants. The competition wants every participant to group by the visitorId and finds the total rmse in each visitorId.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b8857058494f69467d12c7ed65914dc8/href">https://medium.com/media/b8857058494f69467d12c7ed65914dc8/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*RPwO_mxaX1KaZqfiRguPrA.png" /></figure><p>Until now, we did not yet finish the final part because we did not store the result to the CSV and submit.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/709592cb67115f7bcc93be446a9b0349/href">https://medium.com/media/709592cb67115f7bcc93be446a9b0349/href</a></iframe><p>I suggest you check whether the test_pred has the correct shape because the Kaggle only accepts the right shape in the CSV.</p><h3>Conclusion</h3><p>Thank you for coming to the conclusion with me. I hope you could know the means or ways I explained. In the next article, I would review the mistakes in my whole data science project. I make a lot of wrong things in this process. And in this article, I want to tell you that you do not be afraid of machine learning algorithms because you could do well by using Python complete libraries(ex. Scikit Learn, Tensorflow, Keras). Don’t be scared. Just do it!</p><p>I welcome feedback and criticism. You can reach me by sending to my email jacky308082@gmail.com.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24ff1cece9b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Analytics Customer Revenue Prediction -Kaggle challenge (Part Two)]]></title>
            <link>https://ychung38.medium.com/google-analytics-customer-revenue-prediction-kaggle-challenge-part-two-4adda019dbe9?source=rss-a81b27e7e0a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/4adda019dbe9</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Yu Wei Chung]]></dc:creator>
            <pubDate>Sat, 19 Sep 2020 07:45:35 GMT</pubDate>
            <atom:updated>2020-09-19T07:45:35.699Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D2i38W9lZ6o7K10oViD31A.jpeg" /><figcaption>Ｍy hometown/ <a href="https://blog.hotelscombined.com.tw/top3-%E8%8A%B1%E8%93%AE-%E4%BD%8F%E5%AE%BF-%E6%8E%A8%E8%96%A6/">source</a></figcaption></figure><p>It is time to continue to explain this project. In this series of articles, we are walking through implementing a machine learning workflow by taking part in kaggle competition to see how the individual techniques come together. The following steps are <strong>Exploratory Data Analysis </strong>and <strong>Feature Engineering</strong>. I would explain how to use these two parts properly and efficiently.</p><p><a href="https://medium.com/@jacky308082/this-time-i-would-try-to-use-english-to-explain-this-whole-project-4a26c9294ccf">In the first post</a>, I introduced the two steps that contain <strong>Understand the problem and data descriptions </strong>and<strong> data cleaning</strong>. All you prepared for the first post is used to build the right exploratory data analysis and feature engineering. I would discuss these later.</p><p>I welcome feedback and criticism. You can reach me by sending to my email ychung38@asu.edu.</p><h3>Exploratory Data Analysis</h3><p>Before you do the feature engineering, you would not know how to combine these features because you do not know the relationship between the features. However, you could build the feature&#39;s sense by exploring data. I would draw a lot of figures to see what the features might affect the target (the goal in this competition).</p><p>By the way, this problem of the competition is a regression problem, because the target is continuous integers. We should take this problem type in our minds.</p><p>Ok! Let’s start to explore the data.</p><h4>1. One to One</h4><p><strong>Channel to Revenue</strong></p><p>Channel: The channel via which the user came to the Store.</p><p>At the start, we could see how channels affects the revenue. Because the channel is a categorical data, I would draw the bar plot to describe the relationship between channel and revenue</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3SmV_hKCsIPuADrh2bgSwg.png" /></figure><p>As you can see, we could depend on different channels to observe different revenue. This is huge progress because I knew the category could affect the revenue efficiently, so as other categorical features.</p><p><strong>Operation System to Revenue</strong></p><p>It is the as same process as the last implementation. Let me draw the plot directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3SmV_hKCsIPuADrh2bgSwg.png" /></figure><p><strong>Browser to Revenue</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*86lSlrHE8H1-MOtcqRyQcg.png" /></figure><p><strong>Campaign to Revenue</strong></p><p>Maybe it is not very beautiful picture, but I think it is ok to distinguish.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Nxob-lPTngyT-6gNA8INA.png" /></figure><p>Same methods, same impact. This is great to build insight by implementing the exploratory data analysis.</p><h4>2. Build a Pivot table</h4><p>When I implement the one-one part, I would think to put the two features together and see the relationship between two features and the target. Before getting into the feature engineering part, I think this step is the most important part because you would know how deep the features are in this phase. Therefore, I strongly recommend putting a lot of effort into this part to observe the connections among all features. Here I would implement some pivot tables to let you understand what I mean.</p><h4>1. country and source to revenue</h4><p>I would pivot the ‘country’ and ‘source’ see the target by summing. Let’s see!</p><pre>pd.pivot_table(train_df,columns=&#39;geoNetwork.country&#39;,index=&#39;trafficSource.source&#39;,values=&#39;totals.transactionRevenue&#39;,aggfunc=&#39;sum&#39;).plot(kind=&#39;bar&#39;,stacked=True,logy=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FpDGp6o7PVcfdbTxycH69Q.png" /></figure><p>You would wonder what the ‘dark gray’ country is, and it is ‘United States’. Apparently, combine different sources and the different countries would make different revenue. By discovering this relationship, I would merge these two features because I predict they would make an impact on predicting the revenue.</p><p>2. device and channel to revenue</p><p>This is my second example to explain how to build new features that would impact the prediction. In the same way, I hit upon the good idea of considering the channel might be different by using a different devices, so I make the pivot table.</p><pre>pd.pivot_table(train_df,index=&#39;device.r&#39;,columns=&#39;channelGrouping&#39;,values=&#39;totals.transactionRevenue&#39;,aggfunc=&#39;sum&#39;).plot.bar(logy=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-4NQM0cQ1Z-sby_5MPfY1A.png" /></figure><p>Again! Different devices that are accompanied by different channels would make different revenues.</p><p>In the next part, I just used this insight to implement feature engineering. It is usually an efficient but time-consuming part.</p><h3>Feature Engineering</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb384c7f575526f860b46a361904ac61/href">https://medium.com/media/fb384c7f575526f860b46a361904ac61/href</a></iframe><p>This is how the feature engineering worked! At the beginning of thinking about how to deal with these features, I search the proper results in Kaggle kernels and find a lot of good feature engineering. This is an entry-level of feature engineering. We could do a hundred of features by exploring the original features. But here, I think it is enough to bring the machine learning models into the next post.</p><h3>Conclusion</h3><p>This post I introduced the <strong>exploratory data analysis </strong>and <strong>feature engineering</strong>. This is not like other parts of data science methods because this would need you to find some innovative thoughts and insights. Hope this part would help you make a progress in data science.</p><p>I welcome feedback and criticism. You can reach me by sending to my email ychung38@asu.edu.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4adda019dbe9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Analytics Customer Revenue Prediction -Kaggle challenge (Part One)]]></title>
            <link>https://ychung38.medium.com/google-analytics-customer-revenue-prediction-kaggle-challenge-part-one-48a47d689e5f?source=rss-a81b27e7e0a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/48a47d689e5f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Yu Wei Chung]]></dc:creator>
            <pubDate>Wed, 16 Sep 2020 12:24:01 GMT</pubDate>
            <atom:updated>2020-09-16T12:26:42.267Z</atom:updated>
            <content:encoded><![CDATA[<p>This time, I would try to use English to explain this whole project. I might make a lot of mistakes in grammar or words. If you detected anything wrong in these series posts. Please tell me. I sincerely want to improve my English skill.</p><p>Ok! Let’s start it! I took part in a competition in <a href="https://www.kaggle.com/">Kaggle</a>. This competition is called <a href="https://www.kaggle.com/c/ga-customer-revenue-prediction">Google Analytics Customer Revenue Prediction</a>. This competition wants you to predict the revenue that consumers buy after they entered into the google merchandise website. When you entered into google merchandise store, Google would record everything you do on the website. Ex. visited, device.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NoduoTcBVLMJAvBnKpqPUA.png" /></figure><p>Every competition has its own metrics. You have to achieve the highest/lowest score to win the champion. And this competition asks the participant to find the lowest rmse(root mean squared error). If you do not know how the rmse works, you could see the <a href="https://en.wikipedia.org/wiki/Root_mean_square">wiki</a>. I scored the 1.4360, and ranked 949/2579</p><p>The general approach to a Kaggle competition is:</p><ol><li><strong>Understand the problem and data descriptions</strong></li><li><strong>Data cleaning/exploratory data analysis</strong></li><li><strong>Feature engineering/feature selection</strong></li><li><strong>Model optimization</strong></li><li><strong>Interpretation of results</strong></li></ol><p>The series would have three posts to go through the whole project. This post would contain an <strong>understanding of the problem and data descriptions</strong> and <strong>Data cleaning/exploratory data analysis. </strong>You might think it is the easiest part of a data science project. But no. The data scientist usually should pay the most attention here. If you misunderstood the question and misload the dataset, problems would arise. What makes matters worse, it would waste your time to do the afterward process. So, getting ready and following my steps!</p><h3>Understand the problem</h3><p>As I say the problem above, the problem of this competition asked you to predict the revenue in a certain period and certain customers. In this competition, you have a total of 12 features and 1 target(revenue). If you saw the data set, you would discover some features that were dicts. In these dicts, the contents contain several features. So the whole features number would surpass sixty more. But don’t worry, I would handle it. And the metrics in this competition is rmse(root mean squared error)</p><p>First, I would import the libraries I would use in the future.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1c4a758e9f8e083b1904db984b7046d0/href">https://medium.com/media/1c4a758e9f8e083b1904db984b7046d0/href</a></iframe><p>There are many libraries you would use in the project. Some libraries might not be used in a normal another project, but you must form a habit use these libraries. They are essential when you try to make the code clean and wonderful.</p><p>At the first, I do not know how to deal with the dict of the features, After I see an article in Kaggle, he gives the solution to the problem.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ccfa02d7a97954fb6754763005d446c/href">https://medium.com/media/1ccfa02d7a97954fb6754763005d446c/href</a></iframe><p>That is excellent to solve the tough question. Then we could start to preprocess the data. First, I inspect if the train set and test set have the same numbers of columns.</p><pre>set(train_df.columns).difference(test_df.columns)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*cprond0myAz3Tj--39amSg.png" /></figure><p>I would try to find what features are unique and the ratio of unique in every feature.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/117cc61c18edf5da4e7f88c3b6895947/href">https://medium.com/media/117cc61c18edf5da4e7f88c3b6895947/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*s8Sc_cnRpxyD7RjKqIkmyg.png" /></figure><p>You could see these features that contain lots of unique categories. And then, The features that only have one category should be removed because they just have a category. We could distinguish how one category of features affects the target, so I would remove them.</p><pre>drop=[i for i in train_df if train_df[i].nunique(dropna=False)==1]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*DSRkBkYYRuWj97EkIRiLzA.png" /></figure><p>The picture indicated what columns I would drop.</p><pre>#&#39;trafficSource.campaignCode&#39; in train_df, not in test_df<br>train_df=train_df.drop(drop+[&#39;trafficSource.campaignCode&#39;],axis=1)<br>test_df=test_df.drop(drop,axis=1)<br>gc.collect()</pre><p>Next, I came to deal with the missing data. Missing data makes machine learning hard to train the model. In this phase, I just see the ratio of missing data in the train and test set.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8430a355da8d560301f45de8e1b03e88/href">https://medium.com/media/8430a355da8d560301f45de8e1b03e88/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*dqFDyguyuncig2Tz2teckw.png" /></figure><p>You could see the missing ratio of features and do nothing right now. The unique dataframe represents many features that have plenty of categories. Many features contain useless categories. Like ‘trafficSource.source’, the feature is consists of a lot of URL.</p><pre>train_df[train_df[&#39;trafficSource.source&#39;].str.contains(&#39;google&#39;)][&#39;trafficSource.source&#39;].value_counts()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*ek0zjMzzo8oK1ghnbJ5ccg.png" /></figure><p>You could see many ‘google’, but some ‘google’ could not make enough impact because they did not contain many quantities, so I try to merge all the categories that are less than 1000 in on features</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9332c5998804f8df4ca2b45bdde2418b/href">https://medium.com/media/9332c5998804f8df4ca2b45bdde2418b/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*Kg2Ov7lsdAlBPseFLHAffA.png" /></figure><p>I solve the problem that many features have useless categories.</p><h3>Conclusion</h3><p>It is happy to be here. I appreciated your read this article. I preprocess the data and make the important process before the deal with the feature engineering and feature selection. I am very excited that I wrote the first post in English. Thank you again!</p><p>I welcome feedback and criticism. You can reach out to me by sending the email to my email ychung38@asu.edu.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48a47d689e5f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>