<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tetaeho on Medium]]></title>
        <description><![CDATA[Stories by Tetaeho on Medium]]></description>
        <link>https://medium.com/@tetaeho?source=rss-35d06e9bd1e6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*12iXAxYNPztjfxAR</url>
            <title>Stories by Tetaeho on Medium</title>
            <link>https://medium.com/@tetaeho?source=rss-35d06e9bd1e6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:23:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tetaeho/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Setup server using old computers part 2 -Basic docker settings]]></title>
            <link>https://medium.com/@tetaeho/setup-server-using-old-computers-part-2-basic-docker-settings-b432dc506aad?source=rss-35d06e9bd1e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/b432dc506aad</guid>
            <category><![CDATA[root-priviledges]]></category>
            <category><![CDATA[permission-set]]></category>
            <category><![CDATA[ubuntu-server]]></category>
            <category><![CDATA[docker-installation]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Tetaeho]]></dc:creator>
            <pubDate>Tue, 27 Sep 2022 06:44:02 GMT</pubDate>
            <atom:updated>2022-09-27T06:44:02.062Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*HmSrTrFC77lvKTzo3YdD3w.png" /></figure><h3>Overview</h3><p>In this post, we will learn about the basic settings of Docker. In my case docker will be used to run code on data analysis or deploy server and run apache or nginx.</p><h3><strong>Docker</strong></h3><p>Docker is an open source project that runs and manages as a container using process isolation techniques. It is a container technology that does not virtualize the operating system, it is lighter than a virtual machine, and it is good to run several services on one server.</p><p>For security, even if a service is stolen, it is an isolated structure that does not easily affect the original server, so the advantages of virtualization can be utilized to a large extent.</p><h4>Installation</h4><p>Execute following commands in terminal</p><pre>sudo apt-get update</pre><pre>sudo apt-get install ca-certificates curl gnupg lsb-release</pre><pre>sudo mkdir-p/etc/apt/keyrings</pre><pre>curl -fsSL <a href="https://download.docker.com/linux/ubuntu/gpg">https://download.docker.com/linux/ubuntu/gpg</a> | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg</pre><h4>permission settings</h4><p>Basically, when using Docker commands, you must first enter <strong>sudo</strong> to obtain root privileges. However, in this case, you may not be able to access it using vscode. Therefore, set the permissions so that you can use docker even without root privileges.</p><pre>sudo groupadd docker<br>sudo usermod -aGdocker $USER<br>newgrp docker<br>docker ps</pre><h4>Reference</h4><ul><li><a href="https://en.wikipedia.org/wiki/Docker_(software)">https://en.wikipedia.org/wiki/Docker_(software)</a></li><li><a href="https://namu.wiki/w/Docker">https://namu.wiki/w/Docker</a></li><li><a href="https://www.docker.com/">https://www.docker.com/</a></li><li><a href="https://docs.docker.com/engine/install/ubuntu/">https://docs.docker.com/engine/install/ubuntu/</a></li><li><a href="https://docs.docker.com/engine/install/linux-postinstall/">https://docs.docker.com/engine/install/linux-postinstall/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b432dc506aad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setup server using old computers part 1 -Basic Ubuntu settings]]></title>
            <link>https://medium.com/@tetaeho/setup-server-using-old-computers-part-1-69e2d4b6f4a9?source=rss-35d06e9bd1e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/69e2d4b6f4a9</guid>
            <category><![CDATA[ip-address]]></category>
            <category><![CDATA[ubuntu-server]]></category>
            <category><![CDATA[rufus]]></category>
            <category><![CDATA[ssh]]></category>
            <category><![CDATA[hard-drive]]></category>
            <dc:creator><![CDATA[Tetaeho]]></dc:creator>
            <pubDate>Thu, 22 Sep 2022 06:46:50 GMT</pubDate>
            <atom:updated>2022-09-27T06:07:41.027Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Chhx0AjoXyrXufuR.png" /><figcaption>Image source: <a href="https://www.onlinecmag.com/wp-content/uploads/2015/01/how-to-turn-your-old-pc-into-a-web-server.png">https://www.onlinecmag.com/wp-content/uploads/2015/01/how-to-turn-your-old-pc-into-a-web-server.png</a></figcaption></figure><h3><strong>Overview</strong></h3><p>In this post, we will go through basic server settings. Settings will include install Linux/docker/ssh and fix the IP when using the router and how to permanently mount the hard drive. After this setting, you can connect to your server using an ssh program such as putty or bitvise.</p><h3>linux installation(Ubuntu CLI)</h3><p>First, install Linux on the server computer. In my case I installed ubuntu cli. The link for installation is as follows, and you can create an installation USB using rufus. Also the files on the existing server will be deleted, so be sure to make a backup.</p><p>Download <a href="https://ubuntu.com/download/server">Ubuntu</a></p><p>Download <a href="https://rufus.ie/en/">Rufus</a></p><h3>Linux basic program installation</h3><p>Once Ubuntu is installed, update, upgrade, and nano will be installed. Nano is a program that creates/edits documents, and you can use vim instead.</p><p>Execute the commands below in order</p><pre>sudo apt update<br>sudo apt upgrade<br>sudo apt install nano</pre><h3>SSH settings</h3><p><strong>Install SSH</strong></p><pre>sudo apt install openssh-server</pre><p><strong>Check SSH status</strong></p><pre>sudo systemctl status ssh</pre><p><strong>problem 1: If SSH is not running</strong></p><pre>sudo systemctl enable ssh<br>sudo systemctl start ssh</pre><p><strong>problem 2: In case of firewall problem</strong></p><pre>sudo ufw allow ssh</pre><h3>Setup static IP</h3><p>The server connected to the router may have a problem that the ip address (private ip) is changed every time it is restarted, and in this case, the problem of not being able to connect to the server occurs. Therefore, it is necessary to fix the IP address.</p><p><strong>Check Iface value and gateway info</strong></p><pre>route -n</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*Jzy_qkQ-338Q3HEWvMRCzA.jpeg" /></figure><p><strong>Modify the netplan configuration file</strong></p><p>Name of .yaml file could be different</p><pre>sudo nano /etc/netplan/01-network-manager-all.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*0e263t_-KDh_lcHI8eYmIQ.png" /></figure><p>Comment out all existing information and change it’s contents</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*hG-g60i0_8RmNPVs6ZLfzQ.png" /></figure><pre>network:<br>   ethernets:<br>     ens33:  # Iface value<br>       addresses: [172.16.230.134/24]  # Enter the IP address to be fixed.<br>       gateway4: 172.16.230.2  # Gateway information<br>       nameservers:<br>         addresses: [168.126.63.1,8.8.8.8] # Enter the desired name server addresses in the desired order, separated by commas.<br>   version: 2</pre><p><strong>Apply modified configurations</strong></p><pre>sudo netplan apply</pre><h3>Mount hard drive permanently</h3><p>If you add a hard drive to Linux computer, you need to mount it everytime you reboot. To solve this problem, I will explain how to permanently mount new hard drive.</p><p><strong>Check which drive to mount</strong></p><pre>sudo fdisk -l</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*B5rD2nN7Pl2cQkGuQJ4ECQ.jpeg" /></figure><p><strong>Check if it’s UUID</strong></p><pre>sudo blkid</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*-BOUZ1RzFagUyRak2GIl0A.jpeg" /></figure><p><strong>Permanent mount configuration</strong></p><pre>sudo nano /etc/fstab</pre><p>Add hard drive values</p><pre>&lt;disk name or uuid&gt;  &lt;mount point&gt;  &lt;file system&gt;  &lt;mount option&gt;  &lt;dump option&gt;  &lt;check fsck or not&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9xEpX1U-u4Ky1dXE-xSW5A.jpeg" /></figure><p><strong>Reboot</strong></p><pre>sudo reboot</pre><h3>Wrap up</h3><p>In this post, I explained the basic server configuration. If the purpose of building a server is not development, you can use an ssh program such as putty. In the next post, we will learn about basic Docker configuration.</p><h3>References</h3><ul><li><a href="https://codechacha.com/ko/ubuntu-install-openssh/">https://codechacha.com/ko/ubuntu-install-openssh/</a></li><li><a href="https://enowy.tistory.com/33">https://enowy.tistory.com/33</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69e2d4b6f4a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setup server using old computers part 0]]></title>
            <link>https://medium.com/@tetaeho/setup-server-using-old-computers-part-0-f2d0259516a?source=rss-35d06e9bd1e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2d0259516a</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[remote-development]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[vscode]]></category>
            <category><![CDATA[servers]]></category>
            <dc:creator><![CDATA[Tetaeho]]></dc:creator>
            <pubDate>Fri, 16 Sep 2022 11:22:59 GMT</pubDate>
            <atom:updated>2022-09-16T11:22:59.497Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MAG9gAxKhktOzXg17-iphA.jpeg" /></figure><p>Anyone who uses a computer will buy a new computer due to lack of specifications or malfunction. In my case, I have been thinking about whether there is a way to use the computer I used before. At the same time, I was preparing to build a server for more convenient development in data analysis. After a month of preparation, I was able to configure the server I needed, and I would like to share little by little about how I was able to configure it.</p><p><strong>Server Configuration</strong></p><p>In my case, three computers were used to configure the server, and Ubuntu cli was installed on each server, and codes were executed and data was shared using docker and SSH.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*LM0oMvIVLk5bRy0a_WdWcw.png" /></figure><p>The server configuration is as follows.</p><p>Server 1 collects and stores data. The second server analyzes the data of the first server and stores the result again in the first server. Server 3 provides a web service to deploy the results stored in server 1.</p><p>The three servers are connected to the Internet through the same router, and a local computer can access each server from an external network. The local computer accesses the container of each server to write and execute the code required for data collection/analysis/deployment.</p><p><strong>Wrap up</strong></p><p>The server is configured as above, and I am very satisfied with it. Of course, there are various cloud services such as AWS, GCP, and Azure, but building a server yourself can learn a lot and can be a way to save money.</p><p>You don’t necessarily have to use three computers like me, just one computer is enough to configure the server, and I hope that the article I will write in the future will be helpful to those who plan to configure the server.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2d0259516a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimize sorting operation in logistics]]></title>
            <link>https://medium.com/@tetaeho/optimize-sorting-operation-in-logistics-e1b9c1994bab?source=rss-35d06e9bd1e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1b9c1994bab</guid>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[udacity-nanodegree]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[logistics]]></category>
            <dc:creator><![CDATA[Tetaeho]]></dc:creator>
            <pubDate>Sun, 11 Sep 2022 09:57:14 GMT</pubDate>
            <atom:updated>2022-09-11T09:57:14.106Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post, we will explain how we can optimize the block sorting operation using Reinforcement Learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*9f-yPltMflx_JhbJMnJx7Q.png" /><figcaption>Source: <a href="https://www.marineinsight.com/maritime-law/types-of-container-terminals-on-basis-of-ownership/">https://www.marineinsight.com/maritime-law/types-of-container-terminals-on-basis-of-ownership/</a></figcaption></figure><h3>Project overview</h3><p><strong>Problem Statement</strong></p><p>In many industrial sites, raw materials/intermediate materials/finished products are stacked and stored. They range from light products that people can lift, such as products from retailers such as supermarkets, to large and heavy items such as containers or steelplates. The products stored in this way have a set release date or release order, so the product (hereinafter, the block) is pre-arranged for fast shipping. Container terminals and steel stockyards face this problem, and it is known that optimization is very difficult as it is an np-hard problem. Also, for many companies, the question arises as to whether humans are optimal to perform the sorting the blocks themselves. The quicker these tasks are done, the more money the company can save, so we want to solve this problem.</p><p><strong>Project goal</strong></p><p>The main purpose is to arrange each block in the order of shipping to create a state in which all blocks can be shipped without additional sorting operation. And since the minimum number of blocks must be moved to create such a state, success or failure can be determined by comparing the number of movements.</p><h3>Analysis</h3><p><strong>Select optimization method</strong></p><figure><img alt="Source: https://www.kdnuggets.com/2018/03/5-things-reinforcement-learning.html" src="https://cdn-images-1.medium.com/max/700/1*sRcT3MT7wsurUDtBK3k8fw.jpeg" /><figcaption>Basic idea of Reinforcement Learning</figcaption></figure><p>One of the goals of this project is to achieve better results than human decision-making. For this, I decided to apply Reinforcement Learning(hereinafter, the RL), and to utilize the data generated through simulation. It has been proven that RL can produce better results than humans through algorithms such as AlphaGo.</p><p><strong>Data for RL</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*06E70wFMo1YGruksi9YDMg.jpeg" /><figcaption>Example of piled blocks in 2D array</figcaption></figure><p>The data for determining the movement of blocks will be in the form of piles of blocks. If there are 3 stacks and the maximum number of blocks that can be piled is 3, it will have the form of 3*3. Due to these conditions, the form in which the blocks are piled can be expressed as a 2-dimensional array. I decided to use this 2d array as the state in RL.</p><p><strong>State</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*5UseVU-DEi3No5L9lN_eaA.png" /><figcaption>Example of state</figcaption></figure><p>For the natural number of each state, it is shipping sequence of blocks. So block 1 is shipped out first and block 6 will go last. And 0 is an empty space. In the above example, since blocks 6 and 5 are on top of blocks 2 and 4, they cannot be shipped out in order, and cannot be considered to be arranged in the order of shipping.</p><p><strong>Action</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*28Y8cXl_Zh_94s71DT-auw.png" /><figcaption>Example of action(move block from stack 1 to stack 3)</figcaption></figure><p>The movement of each block is an action, and there are several constraints.<br>1. You can only move one block at a time<br>2. Blocks can only move from stack to stack<br>3. If there are maximum number of blocks in the stack to be piled, blocks cannot be piled in that stack</p><p><strong>Reward</strong></p><p>The reward can be thought of as one of the important parameters of RL that quantifies the goal. In this project, the goal is ‘a state in which all blocks can be shipped without additional sorting operation’. Therefore, the reward was made to be granted only when this state was reached.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*iV9xpDFJvMxpEih2w7cq9g.png" /><figcaption>Example of states that can be rewarded</figcaption></figure><p>Therefore, as in the image above, various states exist as the rewarded states. It is also called the terminal state because the block does not need to move any more when it reaches the rewarded state.</p><p><strong>Algorithms and Techniques</strong></p><p>The RL algorithm I will use is Q-learning. An algorithm developed by Google DeepMind showed better results than humans in many classic games. In addition, as the number of states is large, we decided to use an artificial neural network because it was judged that it was difficult to learn with a Q-table, so DQN(Deep Q-Network)was finally applied.</p><p>Since the state is a two-dimensional array, the artificial neural network uses a CNN and a fully connected layer. When the state passes through the artificial neural network, the Q-value corresponding to each action is calculated. At this time, the next state is selected by selecting and executing the action with the largest Q-value. When this method is repeated to reach the terminal state, one episode is finished.</p><h3>Methodology</h3><p><strong>Data Preprocessing</strong></p><p>Data pre-processing is not required as the data to solve this problem is conducted on the premise that it is perfectly observable.</p><p><strong>Implememtation</strong></p><p>There are three main things that need to be implemented in order to apply reinforcement learning to the block ordering task.<br>1. Model block sorting operation for simulation<br>2. Reinforcement Learning Algorithms<br>3. Artificial Neural Network</p><p><strong>Refinement</strong></p><p>One of the important parameters in DQN is the ratio of exploration/exploitation. Exploration is choosing a random action, and exploitation is a method of deciding an action based on training. If there is little exploration at the beginning of training, it is difficult to optimize because various states cannot be experienced. And if there is little exploitation at the end of training, even with enough training random actions are selected.</p><p>In this project, we are going to try various values to find the most appropriate exploration/exploitation ratio and give the best results.</p><p><strong>Initial state</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*ALOtZKVu971vY9thRHaGSg.png" /><figcaption>Initial state</figcaption></figure><p>We decided to train the RL agent to reach the terminal state by determining the above state as the initial state. The initial state is in the form of 4*4, with a total of 11 blocks and 3 shipment orders.</p><p><strong>Neural Network</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*LFtYc32IjCb_Hq_ACSI1Kw.png" /><figcaption>Construction of artificial neural network</figcaption></figure><p>For the artificial neural network, three Convolutional Neural Networks (CNNs) were used, and two FC (Fully Connected) layers were used.</p><p><strong>Parameters for neural network</strong></p><ol><li>Learning rate: 1-e5</li><li>CNN filter size: 128</li><li>Kernel sizes: [1, 1] or [2, 2]</li><li>Stride: [1, 1]</li><li>Batch size: 64</li></ol><p><strong>Parameters for RL</strong></p><ol><li>Replay memory size: 50,000</li><li>Discount factor: 0.95</li><li>Maximum actions per episode: 500</li><li>Training period: 20</li><li>Target network update period: 60</li></ol><p><strong>Test result</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*Zwz6W_TiwQ9Jnfer7Boobw.png" /><figcaption>Result of average steps and average success rate</figcaption></figure><p>The graph above shows the average step (orange line) and the average number of episodes to reach the terminal state (blue line) for every 500 episodes. About 200,000 episodes were performed, and it seems to converge from 170,000 episode. In the latter half of learning, it was found that the terminal state was reached with an average of 11 to 14 steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/1*d5kmFQa1yOH9sxSW40BgHg.gif" /><figcaption>12 step episode 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/1*y7BSOWK2D62MYcRcwTP_HA.gif" /><figcaption>12 step episode 2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/1*KgmSaYaaQ59V0bZjEh6QSA.gif" /><figcaption>6 step episode(Best result)</figcaption></figure><p>3 images above show the process of reaching the terminal state, showing the most frequent 12 steps and 6 step episodes with the best results.</p><p>You can see the results organized as an Excel file in my <a href="https://github.com/2asyhard/Data-Science-Udacity/tree/main/Capstone%20Project">github channel</a></p><h3><strong>Conclusion</strong></h3><p><strong>Reflection</strong></p><p>In this project, RL was applied to optimize logistics. The goal was to sort various materials and products that are piled up and stored with minimal movement according to the shipping date. For this, a DQN was selected for RL algorithm and an artificial neural network was constructed using CNN and FC. Modeling was also done to simulate the sorting operation.<br>At first, the agent selected a random action with high probability and was able to finish the sorting operation through more than 300 actions. However, as the training was repeated gradually, from more than 170,000 episodes, agent was able to finish with 14 or less actions. In addition, it was possible to see the result of finishing the sorting operation with the minimum estimated action of 6 times.<br>Judging from this, if RL is applied to the sorting opeartion, it is considered that the sorting operation close to the optimum, if not optimal, can be performed.</p><p><strong>Source Code: </strong><a href="https://github.com/2asyhard/Data-Science-Udacity/tree/main/Capstone%20Project">https://github.com/2asyhard/Data-Science-Udacity/tree/main/Capstone%20Project</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1b9c1994bab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Find feature that best predict the Developers salary]]></title>
            <link>https://medium.com/@tetaeho/find-feature-that-best-predict-the-developers-salary-4598c4bdc62a?source=rss-35d06e9bd1e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/4598c4bdc62a</guid>
            <category><![CDATA[patterns]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Tetaeho]]></dc:creator>
            <pubDate>Mon, 11 Jul 2022 12:09:55 GMT</pubDate>
            <atom:updated>2022-07-11T12:12:35.588Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*osldQOUxWE00XVsO.jpg" /><figcaption>Image source: <a href="https://cdn.thenewstack.io/media/2022/06/665ebd4b-world-currency-1024x576.jpg">https://cdn.thenewstack.io/media/2022/06/665ebd4b-world-currency-1024x576.jpg</a></figcaption></figure><p><strong>Overview</strong></p><p>Every year, Stackoverflow runs a survey and publishes the results. I would like to investigate the relationship between the programming language used by developers and salary based on 2017 data.</p><p>The questions I have are:</p><ol><li>How would you rank the most used programming languages among respondents?</li><li>Is there a difference in programming languages between people with high salaries and those with low salaries?</li><li>Which column can best predict salary?</li><li>Can you find a pattern related to salary using the best column in the results of Q3?</li></ol><p><strong>Data summary</strong></p><p>The dataset I collected was a stackoverflow survey 2017, in which over 50,000 developers participated in the survey. However, since about 70% of respondents did not answer their salary, the number of data that can actually be used is about 12,000.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*arcsz9d0qRzTS8FqpuNHTw.png" /><figcaption>Stackoverflow survey 2017 data</figcaption></figure><p>It consists of 150 questions including salary and programming language I’m using. Most of columns data types are object, and 6 columns are numeric data type.</p><p>This will be a great dataset to predict salary.</p><p><strong>How would you rank the most used programming languages among respondents</strong></p><p>The most used programming languages are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*AviyMzYdDyTRfwVxldp0bQ.png" /><figcaption>Most used programming language</figcaption></figure><p>And the least used programming languages are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*g348g2oRT2EcxkRlEJE1bQ.png" /><figcaption>Least used programming language</figcaption></figure><p>It is guessed that there are many developers who perform web development using Javascript or data analysis using SQL.</p><p><strong>Is there a difference in programming languages between people with high salaries and those with low salaries?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*WjaR3xEwEzq5Gc7HuGSe5w.png" /><figcaption>Distribution of Salary</figcaption></figure><p>Salary varies greatly depending on the developer. I sorted the salaries in ascending order and created separate dataframes for the top 20% and bottom 20%. After that, statistics of the programming languages used by the top 20% and the bottom 20% were obtained and compared.</p><p>The ranking of programming languages used by the top 20% is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*KlhNfcOXnghdfrGyhRFUUw.png" /><figcaption>Ranking of programming languages used by the top 20%</figcaption></figure><p>And the ranking of programming languages used by the bottom 20% is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*ZNeWnRbrwwXFbfJSY-q_uA.png" /><figcaption>Ranking of programming languages used by the low 20%</figcaption></figure><p>As can be seen from the results, there is no significant difference in the ranking or ratio of the programming languages used. It can be seen that it is difficult to guess the salary of a developer using only the programming language used for this.</p><p><strong>Which column can best predict salary?</strong></p><p>Now let’s actually use machine learning to find the column that best predicts salary. We will perform machine learning with linear regression and compare the results using the r2 score.</p><p>The results are as follow</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/191/1*Bntj5N02S0Z0zP251POsgg.png" /><figcaption>Rank for r2 score</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*Fa81Fb-asHKBdgfNfvdXVg.png" /><figcaption>Histogram for r2 scores for each columns</figcaption></figure><p>According to the results, the currency receiving the salary was the best predictor of salary.</p><p><strong>Can you find a pattern related to salary using the best column in the results of Q3?</strong></p><p>If the currency column is the best predictor of salary, let’s look for the pattern of currency and salary as we did in question 2.</p><p>This is the currency of the top 20% of developers</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*gj-2JGNBnJmgz3SR9MMxoA.png" /><figcaption>High salary currencies</figcaption></figure><p>And this is the currency of the bottom 20% of developers</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*DzIXRmYSwUjCE2KV0fiohA.png" /><figcaption>Low salary currencies</figcaption></figure><p>This time the pattern is very clear. Most of the top 20% get their salaries in USD, and other currencies are negligible. Conversely, the bottom 20% receive salaries in various currencies.</p><p><strong>Result summary</strong></p><ol><li>The programming languages most frequently used by developers appeared in the order of Javascript, SQL, Java, and C#, and the least used programming languages appeared in the order of Hack, Julia, and Dart.</li><li>I thought there might be a difference in salary depending on the language the developers use, but it turns out that there is almost no difference in the language used by the top 20% and the bottom 20%.</li><li>Linear regression was performed to find the column that predicted the salary best among columns in the dataset, and as a result, Currency could have the best r2 score of 0.44 or higher. Since the value representing the language used is 0.05, it was judged that accurate salary calculation was difficult.</li><li>In the results obtained through linear regression, a pattern was found using the column (Currency) with the best r2 score, and it was found that the top 20% received salary in USD.</li></ol><p>Code link: <a href="https://github.com/2asyhard/Data-Science-Udacity/tree/main/Course%201%20Introduction%20to%20Data%20Science/Project">https://github.com/2asyhard/Data-Science-Udacity/tree/main/Course%201%20Introduction%20to%20Data%20Science/Project</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4598c4bdc62a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[One hot encoding with multiple values]]></title>
            <link>https://medium.com/@tetaeho/one-hot-encoding-with-multiple-values-506a88333aac?source=rss-35d06e9bd1e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/506a88333aac</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[get-dummies]]></category>
            <category><![CDATA[one-hot-encoding]]></category>
            <dc:creator><![CDATA[Tetaeho]]></dc:creator>
            <pubDate>Sun, 10 Jul 2022 09:02:37 GMT</pubDate>
            <atom:updated>2022-07-10T09:02:56.188Z</atom:updated>
            <content:encoded><![CDATA[<h3>About one hot encoding</h3><p>There are two types of data: object and numeric, and the input value must be numeric to be used for machine learning. To use object type data, one hot encoding is used, and pandas provides a function called get_dummies. By using the Get_dummies function, object-type data can be converted as follows, and these data will be used for machine learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/149/1*3CUUupKdYoioRb_PZeWzrQ.png" /><figcaption>Before one hot encoding</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*YR86_Y5Do9HVStbpzSYZgA.png" /><figcaption>After one hot encoding</figcaption></figure><h3>Problems with existing get_dummies</h3><p>The get_dummies function of Pandas can easily perform one hot encoding, but even data containing multiple values are one hot encoded and many columns can be created. For example, in the case of the <em>HaveWorkedLanguage </em>column below, the programming language used is selected repeatedly, but after these data are converted, too many columns are created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/292/1*mmRC_-mpwnoHobLW2IJfqw.png" /><figcaption>Before one hot encoding</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*pgtrzue_W9NfYnROVTlQvQ.png" /><figcaption>After one hot encoding</figcaption></figure><h3>Solution</h3><p>To solve this problem, I made a column by collecting the duplicated values separately so that each respondent’s corresponding column was selected. As a result, the number of dummy was reduced from 3445 to 68.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*hL3rVCi7HpFDQ9AJktnrjQ.png" /><figcaption>After using get_dummies function</figcaption></figure><h3>Difference from the existing method (r2score, time required)</h3><p>In order to compare the results of my method with the existing method when applied to machine learning, I decided to estimate salary from stackoverflow survey 2017 data. Linear regression was used, and it was judged whether the prediction was good or bad based on the r2 score. As a result, there was no significant difference in r2 score. However, in the process of regression, there is a time difference of up to 1/100, so it seems that the time can be greatly reduced if the dummy is obtained according to my method. The code for that project can be found <a href="https://github.com/2asyhard/one-hot-encoding-with-multiple-values">here</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=506a88333aac" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>