<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Akira Takezawa on Medium]]></title>
        <description><![CDATA[Stories by Akira Takezawa on Medium]]></description>
        <link>https://medium.com/@akiratakezawa?source=rss-bc2291698c8b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*fCUqyZ8Os4Sk8ehhjS-0HQ.jpeg</url>
            <title>Stories by Akira Takezawa on Medium</title>
            <link>https://medium.com/@akiratakezawa?source=rss-bc2291698c8b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 06:58:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@akiratakezawa/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to implement Seq2Seq LSTM Model in Keras #ShortcutNLP]]></title>
            <link>https://medium.com/shortcutnlp/how-to-implement-seq2seq-lstm-model-in-keras-shortcutnlp-f55fadd7f812?source=rss-bc2291698c8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/f55fadd7f812</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akira Takezawa]]></dc:creator>
            <pubDate>Sun, 07 Apr 2019 09:13:15 GMT</pubDate>
            <atom:updated>2019-04-07T09:13:15.046Z</atom:updated>
            <content:encoded><![CDATA[<h4>If you got stuck with Dimension problem, this is for you</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*EtZMKLxGuVsP42XnnSiy1Q.png" /><figcaption><a href="https://www.techleer.com/articles/440-keras-deep-learning-for-python/">Keras: Deep Learning for Python</a></figcaption></figure><h3>Why do you need to read this?</h3><blockquote>If you got stacked with seq2seq with Keras, I’m here for helping you.</blockquote><p>When I wanted to implement seq2seq for Chatbot Task, I got stuck a lot of times especially about Dimension of Input Data and Input layer of Neural Network Architecture.</p><p>So Here I will explain complete guide of seq2seq for in Keras. Let’s get started!</p><h3>Menu</h3><ol><li>What is Seq2Seq Text Generation Model?</li><li>Task Definition and Seq2Seq Modeling</li><li>Dimensions of Each Layer from Seq2Seq</li><li>Preprocessing of Seq2Seq (in Chatbot Case)</li><li>Simplest preprocessing of code: which you can use today!</li></ol><h3>1. What is Seq2Seq Text Generation Model?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LYGO4IxqUYftFdAccg5fVQ.png" /><figcaption>Fig A — Encoder-Decoder training architecture for NMT — image copyright@<a href="https://medium.com/u/c3f8c66f5451">Ravindra Kompella</a></figcaption></figure><blockquote>Seq2Seq is a type of Encoder-Decoder model using RNN. It can be used as a model for machine interaction and machine translation.</blockquote><p>By learning a large number of sequence pairs, this model generates one from the other. More kindly explained, the definition of Seq2Seq is below:</p><ul><li>Input: Text Data</li><li>Output: Text Data as well</li></ul><p>And here we have examples of business applications of seq2seq:</p><ul><li>Chatbot (you can find from my <a href="https://github.com/samurainote/chatbot_slack_keras">GitHub</a>)</li><li>Machine Translation (you can find from my <a href="https://github.com/samurainote/seq2seq_translate_slackbot">GitHub</a>)</li><li>Question Answering</li><li>Abstract Text Summarization (you can find from my <a href="https://github.com/samurainote/Text_Summarization_using_Bidirectional_LSTM">GitHub</a>)</li><li>Text Generation (you can find from my <a href="https://github.com/samurainote/Text_Generation_using_GRU">GitHub</a>)</li></ul><p>If you want more information about Seq2Seq, here I have a recommendation from <a href="https://www.youtube.com/channel/UCXvHuBMbgJw67i5vrMBBobA">Machine Learning at Microsoft</a> on Yotube:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FZmCzrPVzDQI%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZmCzrPVzDQI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FZmCzrPVzDQI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/516ad2742e3303a692026fc839118ae3/href">https://medium.com/media/516ad2742e3303a692026fc839118ae3/href</a></iframe><p>So let’s take a look at whole process!</p><p>— — — — —</p><h3>2. Task Definition and Seq2Seq Modeling</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*CkeGXClZ5Xs0MhBc7xFqSA.png" /><figcaption><a href="https://www.oreilly.com/library/view/deep-learning-essentials/9781785880360/b71e37fb-5fd9-4094-98c8-04130d5f0771.xhtml">https://www.oreilly.com/library/view/deep-learning-essentials/9781785880360/b71e37fb-5fd9-4094-98c8-04130d5f0771.xhtml</a></figcaption></figure><p>For training our seq2seq model, we will use <a href="https://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html">Cornell Movie — Dialogs Corpus Dataset</a> which contains over 220,579 conversational exchanges between 10,292 pairs of movie characters. And it involves 9,035 characters from 617 movies.</p><p>Here one of the conversations from the data set:</p><pre>Mike: <br>&quot;Drink up, Charley. We&#39;re ahead of you.&quot;</pre><pre>Charley: <br>&quot;I&#39;m not thirsty.&quot;</pre><p>Then we will input these pairs of conversation into Encoder and Decoder. So that means our Neural Network model has two input layer as you can see below.</p><p>This is our Seq2Seq Neural Network Architecture for this time:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5102c53613c09a4473834da56293e923/href">https://medium.com/media/5102c53613c09a4473834da56293e923/href</a></iframe><p>Let’s visualize our Seq2Seq by using LSTM:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*7EUe3SzWJgDbJsI0BISIUQ.png" /><figcaption>copyright Akira Takezawa</figcaption></figure><h3>3. Dimensions of Each Layer from Seq2Seq</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*DRwOwEdpT9wygpfY1DVhVA.jpeg" /><figcaption><a href="https://bracketsmackdown.com/word-vector.html">https://bracketsmackdown.com/word-vector.html</a></figcaption></figure><p>The Black Box for “NLP newbie” is I think this:</p><blockquote>How each layer compiles data and change their Dimension of data?</blockquote><p>To make this clear, I will explain how it works with detail. The Layers can be broken down into 5 different parts:</p><ol><li>Input Layer (Encoder and Decoder)</li><li>Embedding Layer (Encoder and Decoder)</li><li>LSTM Layer (Encoder and Decoder)</li><li>Decoder Output Layer</li></ol><p>Let’s get started!</p><h3>1. Input Layer of Encoder and Decoder (2D-&gt;2D)</h3><ul><li><strong>Input Layer Dimension: 2D (sequence_length, None)</strong></li></ul><pre># 2D<br>encoder_input_layer = Input(shape=(sequence_length, ))<br>decoder_input_layer = Input(shape=(sequence_length, ))</pre><p><em>NOTE: sequence_length is MAX_LEN unified by padding in preprocessing</em></p><ul><li><strong>Input Data: 2D (sample_num, max_sequence_length)</strong></li></ul><pre># Input_Data.shape = (150000, 15)</pre><pre>array([[ 1, 32, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],<br>       [ 123, 56, 3, 34, 43, 345, 0, 0, 0, 0, 0, 0, 0],<br>       [ 3, 22, 1, 6543, 58, 6, 435, 0, 0, 0, 0, 0, 0],<br>       [ 198, 27, 2, 94, 67, 98, 0, 0, 0, 0, 0, 0, 0],<br>       [ 45, 78, 2, 23, 43, 6, 45, 0, 0, 0, 0, 0, 0]<br>        ], dtype=int32)</pre><p><em>NOTE: sample_num can be a length of training_data (150000)</em></p><ul><li><strong>Output Data: 2D</strong></li></ul><p><em>NOTE: </em>Input()<em> is used only for Keras tensor instantiations</em></p><p>— — — — —</p><h3>2. Embedding layer of Encoder and Decoder (2D-&gt;3D)</h3><ul><li><strong>Embedding Layer Dimension: 2D (sequence_length, vocab_size)</strong></li></ul><pre>embedding_layer = Embedding(input_dim = <strong>vocab_size</strong>,<br>                            output_dim = <strong>embedding_dimension</strong>, <br>                            input_length = <strong>sequence_length</strong>)</pre><p><em>NOTE: </em><strong><em>vocab_size</em></strong><em> is the number of unique words</em></p><ul><li><strong>Input Data: 2D (sequence_length, vocab_size)</strong></li></ul><pre># Input_Data.shape = (15, 10000)</pre><pre>array([[ 1, 1, 0, 0, 1, 0, ...... 0, 0, 1, 0, 0, 0, 0],<br>       [ 0, 0, 1, 0, 0, 1, ...... 0, 0, 0, 0, 0, 0, 1],<br>       [ 0, 1, 0, 0, 0, 0, ...... 0, 0, 1, 0, 0, 0, 0],<br>       [ 0, 1, 0, 0, 0, 1, ...... 0, 0, 0, 1, 0, 1, 0],<br>       [ 0, 0, 1, 0, 1, 0, ...... 0, 0, 1, 0, 1, 0, 0]<br>        ], dtype=int32)</pre><p><em>NOTE: Data should be a group of One-Hot Vector</em></p><ul><li><strong>Output Data: 3D (num_samples, sequence_length, embedding_dims)</strong></li></ul><pre># Output_Data.shape = (150000, 15, 50)</pre><pre>array([[[ 1, 1, 0, 0, ...... 0, 1, 0, 0],<br>        [ 0, 0, 1, 0, ...... 0, 0, 0, 1],<br>         ..., <br>         ..., <br>        [ 0, 1, 0, 0, ...... 1, 0, 1, 0],<br>        [ 0, 0, 1, 0, ...... 0, 1, 0, 0]],</pre><pre>[[ 1, 1, 0, 0, ...... 0, 1, 0, 0],<br>        [ 0, 0, 1, 0, ...... 0, 0, 0, 1],<br>         ..., <br>         ..., <br>        [ 0, 1, 0, 0, ...... 1, 0, 1, 0],<br>        [ 0, 0, 1, 0, ...... 0, 1, 0, 0]],</pre><pre>....,] * 150000      , dtype=int32)</pre><p><em>NOTE: Data is word embedded in 50 dimensions</em></p><p>— — — — —</p><h3>3. LSTM layer of Encoder and Decoder (3D-&gt;3D)</h3><p>The tricky argument of LSTM layer is these two:</p><blockquote>1. return_state:</blockquote><blockquote>Whether to return the last state along with the output</blockquote><blockquote>2. return_sequences:</blockquote><blockquote>Whether the last output of the output sequence or a complete sequence is returned</blockquote><p>You can find a good explanation from <a href="https://machinelearningmastery.com/return-sequences-and-return-states-for-lstms-in-keras/"><strong>Understand the Difference Between Return Sequences and Return States for LSTMs in Keras</strong></a><strong> </strong>by <a href="https://machinelearningmastery.com/author/jasonb/"><strong>Jason Brownlee</strong></a>.</p><ul><li><strong>Layer Dimension: 3D (hidden_units, sequence_length, embedding_dims)</strong></li></ul><pre># HIDDEN_DIM = 20</pre><pre>encoder_LSTM = LSTM(HIDDEN_DIM, return_state=True)    encoder_outputs, state_h, state_c = encoder_LSTM(encoder_embedding)</pre><pre>decoder_LSTM = LSTM(HIDDEN_DIM, return_state=True, return_sequences=True)   <br>decoder_outputs, _, _ = decoder_LSTM(decoder_embedding, initial_state=[state_h, state_c])</pre><ul><li><strong>Input Data: 3D (num_samples, sequence_length, embedding_dims)</strong></li></ul><pre># Input_Data.shape = (150000, 15, 50)</pre><pre>array([[[ 1, 1, 0, 0, ...... 0, 1, 0, 0],<br>        [ 0, 0, 1, 0, ...... 0, 0, 0, 1],<br>         ..., <br>         ..., <br>        [ 0, 1, 0, 0, ...... 1, 0, 1, 0],<br>        [ 0, 0, 1, 0, ...... 0, 1, 0, 0]],</pre><pre>[[ 1, 1, 0, 0, ...... 0, 1, 0, 0],<br>        [ 0, 0, 1, 0, ...... 0, 0, 0, 1],<br>         ..., <br>         ..., <br>        [ 0, 1, 0, 0, ...... 1, 0, 1, 0],<br>        [ 0, 0, 1, 0, ...... 0, 1, 0, 0]],</pre><pre>....,] * 150000      , dtype=int32)</pre><p><em>NOTE: Data is word embedded in 50 dimensions</em></p><ul><li><strong>Output Data: 3D (num_samples, sequence_length, hidden_units)</strong></li></ul><pre># HIDDEN_DIM = 20<br># Output_Data.shape = (150000, 15, 20)</pre><pre>array([[[ 0.0032, 0.0041, 0.0021, .... 0.0020, 0.0231, 0.0010],<br>        [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>         ..., <br>         ..., <br>        [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>        [ 0.0021, 0.0065, 0.0008, .... 0.0089, 0.0043, 0.0024]],</pre><pre>[ 0.0032, 0.0041, 0.0021, .... 0.0020, 0.0231, 0.0010],<br>        [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>         ..., <br>         ..., <br>        [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>        [ 0.0021, 0.0065, 0.0008, .... 0.0089, 0.0043, 0.0024]],</pre><pre>....,] * 150000      ,   dtype=int32)</pre><p><em>NOTE: Data reshaped by LSTM as hidden layer in 20 dimensions</em></p><p><strong>Additional Information:</strong></p><p>If return_state = False and return_sequences = False :</p><ul><li><strong>Output Data: 2D (num_sample, hidden_units)</strong></li></ul><pre># HIDDEN_DIM = 20<br># Output_Data.shape = (150000, 20)</pre><pre>array([[ 0.0032, 0.0041, 0.0021, .... 0.0020, 0.0231, 0.0010],<br>       [ 0.0076, 0.0767, 0.0761, .... 0.0098, 0.0065, 0.0076],<br>         ..., <br>         ..., <br>       [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>       [ 0.0021, 0.0065, 0.0008, .... 0.0089, 0.0043, 0.0024]]<br>       , dtype=float32)</pre><p>— — — — —</p><h3>4. Decoder Output Layer (3D-&gt;2D)</h3><ul><li><strong>Output Layer Dimension: 2D (sequence_length, vocab_size)</strong></li></ul><pre>outputs = TimeDistributed(Dense(VOCAB_SIZE, activation=&#39;softmax&#39;))(decoder_outputs)</pre><p><em>NOTE: TimeDistributedDense</em>layer <em>allows us to apply a layer to every temporal slice of an input</em></p><ul><li><strong>Input Data: 3D (num_samples, sequence_length, hidden_units)</strong></li></ul><pre># HIDDEN_DIM = 20<br># Input_Data.shape = (150000, 15, 20)</pre><pre>array([[[ 0.0032, 0.0041, 0.0021, .... 0.0020, 0.0231, 0.0010],<br>        [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>         ..., <br>         ..., <br>        [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>        [ 0.0021, 0.0065, 0.0008, .... 0.0089, 0.0043, 0.0024]],</pre><pre>[ 0.0032, 0.0041, 0.0021, .... 0.0020, 0.0231, 0.0010],<br>        [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>         ..., <br>         ..., <br>        [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>        [ 0.0021, 0.0065, 0.0008, .... 0.0089, 0.0043, 0.0024]],</pre><pre>....,] * 150000      ,   dtype=int32)</pre><p><em>NOTE: Data reshaped by LSTM as hidden layer in 20 dimensions</em></p><ul><li><strong>Output Data: 2D (sequence_length, vocab_size)</strong></li></ul><pre># Output_Data.shape = (15, 10000)</pre><pre>array([[ 1, 1, 0, 0, 1, 0, ...... 0, 0, 1, 0, 0, 0, 0],<br>       [ 0, 0, 1, 0, 0, 1, ...... 0, 0, 0, 0, 0, 0, 1],<br>       [ 0, 1, 0, 0, 0, 0, ...... 0, 0, 1, 0, 0, 0, 0],<br>       [ 0, 1, 0, 0, 0, 1, ...... 0, 0, 0, 1, 0, 1, 0],<br>       [ 0, 0, 1, 0, 1, 0, ...... 0, 0, 1, 0, 1, 0, 0]<br>        ], dtype=int32)</pre><p>After Data passed this Fully Connected Layer, we use <strong>Reversed Vocabulary</strong>which I will explain later to convert from One-Hot Vector into Word Sequence.</p><p>— — — — —</p><h3>4. Entire Preprocess of Seq2Seq (in Chatbot Case)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*ZX16gz2plU0AgIPDMPOYKA.png" /><figcaption>Creating A Language Translation Model Using Sequence To Sequence Learning Approach</figcaption></figure><p>Before jumping on preprocessing of Seq2Seq, I wanna mention about this:</p><blockquote>We need some Variables to define the Shape of our Seq2Seq Neural Network on the way of Data preprocessing</blockquote><ol><li>MAX_LEN: to unify the length of the input sentences</li><li>VOCAB_SIZE: to decide the dimension of sentence’s one-hot vector</li><li>EMBEDDING_DIM: to decide the dimension of Word2Vec</li></ol><p>— — — — —</p><p><strong>Preprocessing for Seq2Seq</strong></p><p>OK, please put this information on your mind, let’s start to talk about preprocessing. The whole process could be broken down into 8steps:</p><ol><li>Text Cleaning</li><li>Put &lt;BOS&gt; tag and &lt;EOS&gt; tag for decoder input</li><li>Make Vocabulary (VOCAB_SIZE)</li><li>Tokenize Bag of words to Bag of IDs</li><li>Padding (MAX_LEN)</li><li>Word Embedding (EMBEDDING_DIM)</li><li>Reshape the Data depends on neural network shape</li><li>Split Data for training and validation, testing</li></ol><p>— — — — —</p><h3>1. Text Cleaning</h3><ul><li><strong>Function</strong></li></ul><p>I always use this my own function to clean text for Seq2Seq:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0db307620796bb42fe8911d4a61f8b2d/href">https://medium.com/media/0db307620796bb42fe8911d4a61f8b2d/href</a></iframe><ul><li><strong>Input</strong></li></ul><pre># encoder input text data</pre><pre>[&quot;Drink up, Charley. We&#39;re ahead of you.&quot;,<br> &#39;Did you change your hair?&#39;,<br> &#39;I believe I have found a faster way.&#39;]</pre><ul><li><strong>Output</strong></li></ul><pre># encoder input text data</pre><pre>[&#39;drink up charley we are ahead of you&#39;,<br> &#39;did you change your hair&#39;,<br> &#39;i believe i have found a faster way&#39;]</pre><p>— — — — —</p><h3>2. Put &lt;BOS&gt; tag and &lt;EOS&gt; tag for decoder input</h3><ul><li><strong>Function</strong></li></ul><p>&lt;BOS&gt; means “Begin of Sequence”, &lt;EOS&gt; means “End of Sequence”.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/58ff0d385e0835663e64a575e2309ad0/href">https://medium.com/media/58ff0d385e0835663e64a575e2309ad0/href</a></iframe><ul><li><strong>Input</strong></li></ul><pre># decoder input text data</pre><pre>[[&#39;with the teeth of your zipper&#39;,<br>  &#39;so they tell me&#39;,<br>  &#39;so  which dakota you from&#39;],,,,]</pre><ul><li><strong>Output</strong></li></ul><pre># decoder input text data</pre><pre>[[&#39;&lt;BOS&gt; with the teeth of your zipper &lt;EOS&gt;&#39;,<br>  &#39;&lt;BOS&gt; so they tell me &lt;EOS&gt;&#39;,<br>  &#39;&lt;BOS&gt; so  which dakota you from &lt;EOS&gt;&#39;],,,,]</pre><p>— — — — —</p><h3>3. Make Vocabulary (VOCAB_SIZE)</h3><ul><li><strong>Function</strong></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc6f39204d026346bb64d3226172028c/href">https://medium.com/media/bc6f39204d026346bb64d3226172028c/href</a></iframe><ul><li><strong>Input</strong></li></ul><pre># Cleaned texts</pre><pre>[[&#39;with the teeth of your zipper&#39;,<br>  &#39;so they tell me&#39;,<br>  &#39;so  which dakota you from&#39;],,,,]</pre><ul><li><strong>Output</strong></li></ul><pre>&gt;&gt;&gt; word2idx</pre><pre>{&#39;genetically&#39;: 14816,<br> &#39;ecentus&#39;: 64088,<br> &#39;houston&#39;: 4172,<br> &#39;cufflinks&#39;: 30399,<br> &quot;annabelle&#39;s&quot;: 23767,<br> .....} # 14999 words</pre><pre>&gt;&gt;&gt; idx2word</pre><pre>{1: &#39;bos&#39;,<br> 2: &#39;eos&#39;,<br> 3: &#39;you&#39;,<br> 4: &#39;i&#39;,<br> 5: &#39;the&#39;,<br> .....} # 14999 indexs</pre><p>— — — — —</p><h3>4. Tokenize Bag of words to Bag of IDs</h3><ul><li>Function</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/653074ec5ff5bd126a3dde15bdfe56f8/href">https://medium.com/media/653074ec5ff5bd126a3dde15bdfe56f8/href</a></iframe><ul><li><strong>Input</strong></li></ul><pre># Cleaned texts</pre><pre>[[&#39;with the teeth of your zipper&#39;,<br>  &#39;so they tell me&#39;,<br>  &#39;so  which dakota you from&#39;],,,,]</pre><ul><li><strong>Output</strong></li></ul><pre># decoder input text data</pre><pre>[[10, 27, 8, 4, 27, 1107, 802],<br> [3, 5, 186, 168],<br> [662, 4, 22, 346, 6, 130, 3, 5, 2407],,,,,]</pre><p>— — — — —</p><h3>5. Padding (MAX_LEN)</h3><ul><li><strong>Function</strong></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/079977f1df9073fef08c9662eaef13d8/href">https://medium.com/media/079977f1df9073fef08c9662eaef13d8/href</a></iframe><ul><li><strong>Input</strong></li></ul><pre># decoder input text data</pre><pre>[[10, 27, 8, 4, 27, 1107, 802],<br> [3, 5, 186, 168],<br> [662, 4, 22, 346, 6, 130, 3, 5, 2407],,,,,]</pre><ul><li><strong>Output</strong></li></ul><pre># MAX_LEN = 10<br># decoder input text data</pre><pre>array([[10, 27, 8, 4, 27, 1107, 802, 0, 0, 0],<br>       [3, 5, 186, 168, 0, 0, 0, 0, 0, 0],<br>       [662, 4, 22, 346, 6, 130, 3, 5, 2407, 0],,,,,]</pre><p>— — — — —</p><h3>6. Word Embedding (EMBEDDING_DIM)</h3><ul><li><strong>Function</strong></li></ul><p>We use Pretraind Word2Vec Model from Glove. We can create embedding layer with Glove with 3 steps:</p><ol><li>Call Glove file from XX</li><li>Create Embedding Matrix from our Vocabulary</li><li>Create Embedding Layer</li></ol><p>Let’s take a look!</p><ul><li><strong>Call Glove file from XX</strong></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/14458b4c5dbf2824875b01ffae96f304/href">https://medium.com/media/14458b4c5dbf2824875b01ffae96f304/href</a></iframe><ul><li><strong>Create Embedding Matrix from our Vocabulary</strong></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/784d219d425a1ca9fe933c7d980a69be/href">https://medium.com/media/784d219d425a1ca9fe933c7d980a69be/href</a></iframe><ul><li><strong>Create Embedding Layer</strong></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d5226699e42d308e61acd95bc8ddac4/href">https://medium.com/media/0d5226699e42d308e61acd95bc8ddac4/href</a></iframe><p>— — — — —</p><h3>7. Reshape the Data to neural network shape</h3><ul><li><strong>Function</strong></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34b74f952a1be943cd1dc63b72d03771/href">https://medium.com/media/34b74f952a1be943cd1dc63b72d03771/href</a></iframe><ul><li><strong>Input</strong></li></ul><pre># MAX_LEN = 10<br># decoder input text data</pre><pre>array([[10, 27, 8, 4, 27, 1107, 802, 0, 0, 0],<br>       [3, 5, 186, 168, 0, 0, 0, 0, 0, 0],<br>       [662, 4, 22, 346, 6, 130, 3, 5, 2407, 0],,,,,]</pre><ul><li><strong>Output</strong></li></ul><pre># output.shape (num_samples, MAX_LEN, VOCAB_SIZE)<br># decoder_output_data.shape (15000, 10, 15000)</pre><pre>array([[[0., 0., 0., ..., 0., 0., 0.],<br>        [0., 0., 0., ..., 0., 0., 0.],<br>        [0., 0., 1., ..., 0., 0., 0.],<br>        ...,<br>        [1., 0., 0., ..., 0., 0., 0.],<br>        [1., 0., 0., ..., 0., 0., 0.],<br>        [1., 0., 0., ..., 0., 0., 0.]],</pre><pre>..., ], , dtype=float32)</pre><p>— — — — —</p><h3>8. Split Data for training and validation, testing</h3><ul><li><strong>Function</strong></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a8646e80d9bc6ab0d8da9939976d01e/href">https://medium.com/media/5a8646e80d9bc6ab0d8da9939976d01e/href</a></iframe><p>— — — — —</p><h3>References</h3><ul><li><a href="https://chunml.github.io/ChunML.github.io/project/Sequence-To-Sequence/">Creating A Language Translation Model Using Sequence To Sequence Learning Approach</a></li><li><a href="http://www.abigailsee.com/2017/04/16/taming-rnns-for-better-summarization.html">Taming Recurrent Neural Networks for Better Summarization</a></li><li><a href="https://github.com/oswaldoludwig/Seq2seq-Chatbot-for-Keras">oswaldoludwig/Seq2seq-Chatbot-for-Keras</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f55fadd7f812" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shortcutnlp/how-to-implement-seq2seq-lstm-model-in-keras-shortcutnlp-f55fadd7f812">How to implement Seq2Seq LSTM Model in Keras #ShortcutNLP</a> was originally published in <a href="https://medium.com/shortcutnlp">Coldstart.ml</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[#06 Model Validation: Practical Metric Lists only you need to know]]></title>
            <link>https://medium.com/shortcutnlp/06-model-validation-practical-metric-lists-only-you-need-to-know-41e7db16089d?source=rss-bc2291698c8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/41e7db16089d</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akira Takezawa]]></dc:creator>
            <pubDate>Sat, 30 Mar 2019 11:23:00 GMT</pubDate>
            <atom:updated>2019-03-30T11:23:00.536Z</atom:updated>
            <content:encoded><![CDATA[<h4>モデルの評価指標まとめ</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5VxzFMzJO0oKlNn0JTtEyg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/Jd810pf_270?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">rawpixel</a> on <a href="https://unsplash.com/search/photos/measure?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Hola! Welcome to #ShortcutML Series! Cheat Note for everyone!</p><h3>Target is who wanna know …</h3><ul><li>Reason: Model Applying isn’t the end of ML</li><li>Big Picture: Summarize too many validation metrics</li><li>Code: The simplest python code for each preprocessing</li></ul><p>— — —</p><h3>Why you have to read this?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*cUVoSzcBt4jqk-jF780-Nw.png" /><figcaption>Choosing the Right Metric for Evaluating Machine Learning Models — Part 1, <a href="https://medium.com/@aswalin">Alvira Swalin</a></figcaption></figure><p>Before we jump on the main topic, when do we evaluate our model? The answer is not only once. Generally, we use model validation metrics twice in our real Data Science workflow:</p><ol><li>Model Comparison: select the best ML model for your task</li><li>Model Improvement: with tuning hyperparameters</li></ol><p>To get a more clear picture of the difference between these two, let me explain by the workflow of ML implementation. So after you set all the features X for your task y, you might prepare multiple ML models as candidates.</p><p>Then how can you finally choose one for your task? Yes, this is the first point when you use model validation metrics. Scikit-learn provides some shortcut methods to compare models like cross_validation.</p><p>Next, after you choose one particular model with the best accuracy, you will jump on Hyperparameter Tuning part to improve more accuracy and versatility. Here is the second point you’ll use these metrics.</p><p>In this article, I’m trying to make a Cheat Note of Model Evaluation Metrics. So let’s get started!</p><p>— — —</p><h3>Menu</h3><ol><li>Cross-Validation</li><li>Metrics for Regression problem</li><li>Metrics for the Classification problem</li><li>Metrics for Clustering problem</li><li>Additional: Learning Curve Visualization</li></ol><h3>1. Cross-Validation for model comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*2HuBvYfFo_ccqRf3YcECsw.jpeg" /><figcaption>Visual Representation of Train/Test Split and Cross Validation. H/t to my <a href="https://generalassemb.ly/education/data-science-immersive">DSI</a> instructor, <a href="https://medium.com/u/690e8d667b35">Joseph Nelson</a></figcaption></figure><p>The starting point of this why and how we split data is <strong>Generalization</strong>. Because our goal of building a machine learning model is real implementation with unknown data from the future. So we don’t need useless models which are overfitting with past data.</p><p>Therefore the biggest difference between these two methods is the way of handling “training data”. One is fix training data, but another is randomly and diversely picking up training data to create a more generalized model.</p><p><strong>1. Holdout Method</strong></p><pre>from sklearn.model_selection import train_test_split<br>from sklearn.datasets import load_wine<br>wine = load_wine()<br>X, y = wine.data, wine.target<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)</pre><p>2. <strong>Cross-Validation Method</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qfDi_Qf81xbTTk4VFmUtHA.jpeg" /><figcaption>Visual representation of K-Folds. Again, H/t to <a href="https://medium.com/u/690e8d667b35">Joseph Nelson</a></figcaption></figure><pre># Decision Tree Classifieras for estimator<br>from sklearn.tree import DecisionTreeClassifier<br>clf = DecisionTreeClassifier(random_state=0)</pre><p>2–1. cross_val_score: simplest coding method</p><p>We can decide the number of data splitting by a parameter “cv”. Normally 5 is considered as a standard splitting number.</p><pre># X, y = wine.data, wine.target<br>from sklearn.model_selection import cross_val_score<br>scores = cross_val_score(clf, X, y, cv=5)<br>print(scores) # cv = number of splited data<br>print(scores.mean())</pre><pre>&gt;&gt;&gt; array([0.78378378, 0.86111111, 0.88888889, 0.91428571, 0.85294118])<br>&gt;&gt;&gt; 0.86 # Accuracy was 86% (not enough!)</pre><p>2–1. cross_validate: I recommend this customizable one</p><pre>scoring = [&#39;precision_macro&#39;, &#39;recall_macro&#39;]<br>scores = cross_validate(clf, X, y, scoring=scoring, cv=5)<br>print(scores)</pre><pre>&gt;&gt;&gt; {&#39;test_recall_macro&#39;: array([0.76666667, 0.85238095, 0.90079365, &gt;&gt;&gt; 0.91137566, 0.88095238]), <br>&gt;&gt;&gt; &#39;test_precision_macro&#39;: array([0.79878618, 0.86602564, <br>&gt;&gt;&gt; 0.88888889, 0.91851852, 0.87777778]), <br>&gt;&gt;&gt; &#39;score_time&#39;: array([0.00279498, 0.00261092, 0.00165415, <br>&gt;&gt;&gt; 0.00270295, 0.0016489 ]), <br>&gt;&gt;&gt; &#39;fit_time&#39;: array([0.00161314, 0.00124598, 0.00124192, <br>&gt;&gt;&gt; 0.00087595, 0.00107622])}</pre><h3>2. Metrics for Regression</h3><p>TL;DR: In most case, we use R2 or RMSE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*__QeqlLr4nNn00SlxUF68A.jpeg" /><figcaption>answered Apr 18 ’15 at 12:00 <a href="https://stats.stackexchange.com/users/42380/jean-paul">Jean-Paul</a></figcaption></figure><p>I’ll use Boston House Price dataset.</p><pre># Data Preparation<br>from sklearn.datasets import load_boston<br>boston = load_boston()<br>X, y = boston.data, boston.target</pre><pre># Train data and Test data Splitting<br>from sklearn.model_selection import train_test_split<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)</pre><p><strong>Model 1: Linear Regression</strong></p><pre>reg = LinearRegression()<br>reg.fit(X_train, y_train)<br>y_pred1 = reg1.predict(X_test)</pre><p><strong>Model 2: Decision Tree Regressor</strong></p><pre>from sklearn.tree import DecisionTreeRegressor<br>reg2 = DecisionTreeRegressor(max_depth=3)<br>reg2.fit(X_train, y_train)<br>y_pred2 = reg2.predict(X_test)</pre><p>Now we are ready to evaluate our two models and chose one!</p><p><strong>1. R2: Coefficient of Determination</strong></p><pre>from sklearn.metrics import r2_score<br>r2_score(y_test, y_pred1) # Linear Regression<br>r2_score(y_test, y_pred2) # Decision Tree Regressor</pre><pre>&gt;&gt;&gt; 0.693909.. <br>&gt;&gt;&gt; 0.693134.. # Decision Tree Regressor won!</pre><p>when to use:</p><p><strong>2. MSE: Mean Square Error</strong></p><pre>from sklearn.metrics import mean_squared_error<br>mean_squared_error(y_test, y_pred)</pre><pre>&gt;&gt;&gt; 23.873348..</pre><p>when to use:</p><p><strong>3. RMSE: Root Mean Square Error</strong></p><pre>import numpy as np<br>np.sqrt(mean_squared_error(y_test, y_pred))</pre><pre>&gt;&gt;&gt; 4.886036..</pre><p>when to use:</p><p><strong>4. MAE: Mean Absolute Error</strong></p><pre>reg = LinearRegression()<br>reg.fit(X_train, y_train)<br>y_pred = reg.predict(X_test)<br>mean_absolute_error(y_test, y_pred)</pre><pre>&gt;&gt;&gt; 3.465279..</pre><p>when to use:</p><h3>3. Metrics for Classification</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*c52VFuXDilef7EFfhagyYA.jpeg" /><figcaption><a href="https://sklearn.org/modules/svm.html">https://sklearn.org/modules/svm.html</a></figcaption></figure><p><strong>Overall Picture for a classification problem</strong></p><ol><li>One vs. One Classification: e.g. Paid user or free</li><li>One vs. Rest Classification: e.g. Premium member or Paid or free</li></ol><p>I’ll use Iris dataset as a multi-class classification problem.</p><pre># Data Preparation<br>from sklearn.datasets import load_iris<br>iris = load_iris()<br>X, y = iris.data, iris.target</pre><pre># Train data and Test data Splitting<br>from sklearn.model_selection import train_test_split<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)</pre><p><strong>Model 1: SVM</strong></p><pre>from sklearn.svm import SVC<br>clf1 = SVC(kernel = &#39;linear&#39;, C = 0.01)<br>clf1.fit(X_train, y_train)<br>y_pred1 = clf1.predict(X_test)</pre><p><strong>Model 2: Naive Bayes</strong></p><pre>from sklearn.naive_bayes import GaussianNB<br>clf2 = GaussianNB()<br>clf2.fit(X_train, y_train)<br>y_pred2 = clf2.predict(X_test)</pre><p>Now we are ready to evaluate our two models and chose one!</p><p><strong>1. Accuracy:</strong></p><pre>from sklearn.metrics import accuracy_score<br>accuracy_score(y_test, y_pred1)<br>accuracy_score(y_test, y_pred2)</pre><pre>&gt;&gt;&gt; 0.933333.. <br>&gt;&gt;&gt; 0.888888.. # Gaussian Naive Bayes won!</pre><p>when to use:</p><p><strong>2. Precision:</strong></p><pre>from sklearn.metrics import precision_score<br>precision_score(y_test, y_pred1, average=None)<br>precision_score(y_test, y_pred2, average=None)</pre><pre>&gt;&gt;&gt; array([1.  , 0.875     , 0.92307692])<br>&gt;&gt;&gt; array([1.  , 0.88235294, 1.        ]) # again GNB won!</pre><p>when to use:</p><p><strong>3. Recall or Sensitivity:</strong></p><pre>from sklearn.metrics import recall_score<br>recall_score(y_test, y_pred2, average=None)</pre><pre>&gt;&gt;&gt; array([1.        , 1.        , 0.85714286]) # GNB</pre><p>when to use:</p><p><strong>4. F Score:</strong></p><pre>from sklearn.metrics import f1_score<br>f1_score(y_test, y_pred2, average=None)</pre><pre>&gt;&gt;&gt; array([1.        , 0.9375    , 0.92307692]) # GNB</pre><p>when to use:</p><p><strong>5. Confusion Matrix</strong></p><pre>from sklearn.metrics import confusion_matrix<br>confusion_matrix(y_test, y_pred2)</pre><pre>&gt;&gt;&gt; array([[16,  0,  0],<br>           [ 0, 15,  0],<br>           [ 0,  2, 12]]) # GNB</pre><p>when to use:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*o8VhTKudDp_Km8-z1Jx6Rg.jpeg" /><figcaption><a href="https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html">scikit-learn official document</a></figcaption></figure><p><strong>6. ROC: Receiver Operating Characteristic Curve</strong></p><p>If you don’t use OneVsRestClassifier, it doesn’t work…</p><pre>from sklearn.multiclass import OneVsRestClassifier<br>from sklearn.svm import LinearSVC<br>clf = OneVsRestClassifier(LinearSVC(random_state=0))<br>clf.fit(X_train, y_train)<br>y_pred = clf.predict(X_test)</pre><p>Now we will check by ROC Curve.</p><pre>from sklearn.metrics import roc_curve<br>fpr, tpr, thresholds = roc_curve(y_test,  y_pred, pos_label=2)<br>fpr, tpr, thresholds</pre><pre>&gt;&gt;&gt; array([0.  , 0.1 , 0.53333333, 1. ]) # False-Positive Rate<br>&gt;&gt;&gt; array([0., 1., 1., 1.]) # True-Positive Rate<br>&gt;&gt;&gt; array([3, 2, 1, 0]) # thresholds</pre><p>when to use:</p><p><strong>7. AUC: Area Under Curve</strong></p><pre>from sklearn.metrics import auc<br>auc(fpr, tpr)</pre><pre>&gt;&gt;&gt; 0.913333... # auc</pre><p>when to use:</p><p><strong>8. Multi-class logarithmic loss</strong></p><p>It’s a probability. And need to use OneVsRestClassifier.</p><pre># clf = OneVsRestClassifier(LinearSVC(random_state=0))<br>from sklearn.metrics import log_loss<br>y_pred = clf.predict_proba(X_test) # not .predict()<br>log_loss(y_test,  y_pred)</pre><pre>&gt;&gt;&gt; 0.09970990582482485</pre><p>when to use:</p><h3>4. Metrics for Clustering</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*wn8_6xt2b4dGlEdtKLsFLw.gif" /><figcaption><a href="https://aws.amazon.com/jp/blogs/machine-learning/amazon-sagemaker-supports-knn-classification-and-regression/">Amazon SageMaker supports kNN classification and regression</a></figcaption></figure><p>Basically in real clustering task, (I mean unsupervised clustering), we don’t have any method to measure accuracy or precision because nobody knows.</p><p>However as a process of classification task, sometimes we use supervised clustering to know the character of data. (In a real job as well.)</p><p>So I’ll quickly introduce some metrics for supervised clustering, in order to let you know just their existence (priority is very low though).</p><p>OK, I used only features from Iris dataset for a clustering problem.</p><pre>from sklearn.datasets import load_iris<br>iris = load_iris()<br>X, y = iris.data, iris.target</pre><p>As a representative model for a clustering problem, This time I used K-means.</p><pre>from sklearn.cluster import KMeans<br>kmeans = KMeans(n_clusters=3, random_state=0)<br>kmeans.fit(X)<br>y_means = kmeans.predict(X)</pre><p>Now the result of supervised clustering is In y_means.</p><pre># visulaliza result!<br>plt.scatter(X[y_means==0,0],X[y_means==0,1],s=50, c=&#39;orange&#39;,label=&#39;Cluster1&#39;)<br>plt.scatter(X[y_means==1,0],X[y_means==1,1],s=50, c=&#39;navy&#39;,label=&#39;Cluster2&#39;)<br>plt.scatter(X[y_means==2,0],X[y_means==2,1],s=50, c=&#39;green&#39;,label=&#39;Cluster3&#39;)<br>plt.scatter(kmeans.cluster_centers_[:,0], kmeans.cluster_centers_[:,1],s=200,marker=&#39;s&#39;, c=&#39;red&#39;, alpha=0.6, label=&#39;Centroids&#39;)<br>plt.title(&#39;Iris segments&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*ZkVMtFxUKQ_DzbRnUk_sHw.jpeg" /><figcaption>© 2019 akira takezawa</figcaption></figure><p><strong>1. Homogeneity score, Completeness Score, V-measure Score</strong></p><pre>from sklearn.metrics import homogeneity_score, completeness_score, v_measure_score<br>hg = homogeneity_score(y, y_means) <br>co = completeness_score(y, y_means)<br>vm = v_measure_score(y, y_means)<br>print(hg, co, vm)</pre><pre>&gt;&gt;&gt; 0.751485..<br>&gt;&gt;&gt; 0.764986..<br>&gt;&gt;&gt; 0.758175..</pre><h3>5. Additional: Learning Curve Visualization</h3><pre>from sklearn.model_selection import learning_curve<br>from sklearn.model_selection import ShuffleSplit</pre><pre>def plot_learning_curve(clf, title, X, y, ylim=None, cv=None,<br>                        n_jobs=None, train_sizes=np.linspace(.1, 1.0, 5)):<br>    plt.figure()<br>    plt.title(title)<br>    if ylim is not None:<br>        plt.ylim(*ylim)<br>    plt.xlabel(&quot;Training examples&quot;)<br>    plt.ylabel(&quot;Score&quot;)<br>    train_sizes, train_scores, test_scores = learning_curve(<br>        clf, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)<br>    train_scores_mean = np.mean(train_scores, axis=1)<br>    train_scores_std = np.std(train_scores, axis=1)<br>    test_scores_mean = np.mean(test_scores, axis=1)<br>    test_scores_std = np.std(test_scores, axis=1)<br>    plt.grid()</pre><pre>plt.fill_between(train_sizes, train_scores_mean - train_scores_std,<br>                     train_scores_mean + train_scores_std, alpha=0.1,<br>                     color=&quot;r&quot;)<br>    plt.fill_between(train_sizes, test_scores_mean - test_scores_std,<br>                     test_scores_mean + test_scores_std, alpha=0.1, color=&quot;g&quot;)<br>    plt.plot(train_sizes, train_scores_mean, &#39;o-&#39;, color=&quot;r&quot;,<br>             label=&quot;Training score&quot;)<br>    plt.plot(train_sizes, test_scores_mean, &#39;o-&#39;, color=&quot;g&quot;,<br>             label=&quot;Cross-validation score&quot;)</pre><pre>plt.legend(loc=&quot;best&quot;)<br>    return plt</pre><pre>title = &quot;Learning Curves (Decision Tree, max_depth=2)&quot;<br>cv = ShuffleSplit(n_splits=100, test_size=0.2, random_state=0)</pre><pre>clf = DecisionTreeClassifier(max_depth=2, random_state=0)<br>plot_learning_curve(clf, title, X, y, ylim=(0.7, 1.01), cv=cv, n_jobs=4)</pre><pre>title = &quot;Learning Curves (SVM, Decision Tree, max_depth=5)&quot;<br>cv = ShuffleSplit(n_splits=10, test_size=0.2, random_state=0)<br>clf = DecisionTreeClassifier(max_depth=5, random_state=0)<br>plot_learning_curve(clf, title, X, y, (0.7, 1.01), cv=cv, n_jobs=4)</pre><pre>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*hgSDXI8KIK2hMfdpZp2MKw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*8w5fEMgW88A5WwDCpE2i0g.png" /></figure><p>— — — — —</p><h3>References</h3><ul><li><a href="https://rapidminer.com/resource/correct-model-validation/">How to Correctly Validate Machine Learning Models | RapidMiner</a></li><li><a href="https://towardsdatascience.com/supervised-machine-learning-model-validation-a-step-by-step-approach-771109ae0253">Supervised Machine Learning: Model Validation, a Step by Step Approach</a></li><li><a href="https://www.accenture.com/us-en/insight-emerging-trends-machine-learning">Machine Learning and AI Model Validation for Financial Firms | Accenture</a></li><li><a href="https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-model-validation.html">Validate a Machine Learning Model - Amazon SageMaker</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41e7db16089d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shortcutnlp/06-model-validation-practical-metric-lists-only-you-need-to-know-41e7db16089d">#06 Model Validation: Practical Metric Lists only you need to know</a> was originally published in <a href="https://medium.com/shortcutnlp">Coldstart.ml</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Golden Path to become A Full-Stack Data Scientist — Who is needed by in Real Industry.]]></title>
            <link>https://medium.com/@akiratakezawa/the-golden-path-to-become-a-full-stack-data-scientist-who-is-needed-by-in-real-industry-e865e47e3788?source=rss-bc2291698c8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e865e47e3788</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Akira Takezawa]]></dc:creator>
            <pubDate>Wed, 27 Mar 2019 03:24:43 GMT</pubDate>
            <atom:updated>2019-04-03T12:56:35.897Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Golden Path to become A Full-Stack Data Scientist — Who is needed by in Real Industry.</h3><h4>Actual report from 175 days of job shifting to ML field</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LUBN4_IVEO2pkoAJ2qAzPg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/kdNs6NZMaD4?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Falco Negenman</a> on <a href="https://unsplash.com/search/photos/art?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Why do you have to read this?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*PuDcUlhu6ccUV3p493CJ2Q.png" /><figcaption><a href="https://www.glassdoor.com/Salaries/data-scientist-salary-SRCH_KO0,14.htm">from glassdoor.com on 25th of March, 2019</a></figcaption></figure><blockquote>The shortage of data scientists is becoming a serious constraint in some sectors. — <a href="https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century">Data Scientist: The Sexiest Job of the 21st Century</a> from Havard Business Review</blockquote><p>Nowadays <strong>Big Data</strong> is one of the most important key resources to get a competitive advantage in business, especially for IT companies. All of <a href="https://www.quora.com/What-is-GAFA">GAFA</a>, accelerate their business by applying Data Science Technique. Let me explain a little bit more.</p><p>For instance, Google, <strong>Youtube uses Recommend Engine</strong> which will never let you go, because they suggest contents by completely following your taste. <strong>Amazon increases its </strong><a href="https://rnnnews.jp/search/result/?q=gross+merchandise+volume"><strong>Gross Merchandise Volume</strong></a>(GMS) by matching user and product very efficiently. <strong>Facebook shows Advertisements with higher </strong><a href="https://www.mobvista.com/en/blog/what-are-cvr-and-ctr/"><strong>CVR</strong></a> from extracting critical insights from your demographical and behavioral data. (I used their Ads in the previous job and was surprised at how effective it performed.)</p><p>Till now I’ve only mentioned about Internet Giants. However, according to <a href="https://money.cnn.com/interactive/technology/masayoshi-son-profile/index.html"><strong>Masayoshi Son</strong></a> who is Japanese CEO of SoftBank Vision Fund’s which is one of the biggest founds all over the world, <a href="https://asia.nikkei.com/Business/Companies/SoftBank-Vision-Fund-s-investments-hover-near-45bn-in-2018">and investments hover near $45 Billion dollars in 2018</a>, left an insightful comment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GLJGZ3L-U2HVN92fS1LcWg.jpeg" /><figcaption>image from SoftBank’s unicorn hunter</figcaption></figure><blockquote>The impact of Internet is somehow limited in particular domains or industries, like Advertisement, Retail (E-commerce), but AI is different. — Masoyoshi Son</blockquote><p><strong>“Next Big Waves”</strong> in other industries, which is applying data science methods into <strong>Unstructured Data </strong>like Image, Natural Language, Sounds, are coming up. I mean, for example, <strong>Mobility Industry </strong>you can see what Elon Mask does, <strong>Robotics</strong> you see what’s happening in Amazon’s warehouse, <strong>Finance</strong>, <strong>Healthcare</strong> area and so on.</p><p><strong>If you are young and wanna be the biggest winner in your career</strong>, this Data Science or Machine Learning field could be the most possible choice.</p><p>Because from my marketing perspective, a quite simple demand-supply balance analysis, I think some special paid job title like professionals in the Finance field or classic Software Engineer, Business Development position with MBA grad is<strong> already taken by our elders</strong>. And those skills and knowledge are became commoditized. I mean these are very competitive.</p><p>Comparing to them, <strong>Data Science field is way messier and still in the mist.</strong> Then why not invest your career and passion, intelligence into this challenging field? Welcome to this fantastic ML technology with a full of hope!</p><p>In this article, I will focus on h<strong>ow to get a Data Scientist jobs or Machine Learning Engineer posts </strong>who will be needed by real industries, or in other words, who will pass the job hunting process with lesser pain. Let’s get started!</p><p>— — —</p><h3>After you read this, you’ll get:</h3><ul><li><strong>The Shortest Path</strong> towards A Real Data Scientist</li><li><strong>The Best Learning Resource</strong> everyone trust in this area</li><li><strong>The Realistic Possibility</strong> of your career with ML</li></ul><p>— — —</p><h3>Menu</h3><ol><li>What is Data Science for Business?</li><li>Data scientist vs Machine Learning Engineer</li><li>Required Skill for A Full-Stuck Data Scientist</li><li>A Golden DS Learning path for a newbie</li><li>The Secret Bibles towards A Full-Stack Data Scientist</li></ol><p>— — —</p><h3>1. What is Data Science for Business?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SyaztIXrRAPj2vADJPICEA.png" /><figcaption>from <a href="https://www.edureka.co/blog/what-is-machine-learning/">edureka!</a></figcaption></figure><blockquote>Data Science itself is just a Method, Not a Goal in our business. Then definition of it should be explained in much simpler way. — an unknown data scientists</blockquote><p>Data Science itself can never be <strong>a purpose</strong>, if it becomes, it means you already fail. Rather than that, I would say “We just became able to make some <strong>additional or novel values</strong> in the real business which we couldn’t 10 years before”.</p><p>I highly recommend listening to this podcast, <a href="https://www.superdatascience.com/podcast/podcast-one-purpose-data-science-truth-analytics"><strong>SDS 131: The One Purpose to Data Science and The Truth about Analytics</strong></a> from <a href="https://www.superdatascience.com/">SuperDataScience</a>.</p><p>To define Data Science, I wanted to start from their goal and purpose. <strong>The fundamental goals of Data Science</strong> in business is pretty clear. I think it can be simply described as followings:</p><ul><li>To <strong>make more profit</strong> in your business by using data (as a result)</li><li>To understand and <strong>satisfy your customers</strong> by efficiency, better matching</li><li>To <strong>create a new business</strong> or startup by using machine learning</li></ul><h3>The background behind this Game Change</h3><p>In addition, I think it’s very important to understand the reasons why the significance of Data Science is gradually recognized in recent business. It could be the following three:</p><ul><li>An explosion the <strong>Amount of Data by Internet and Smartphone</strong></li><li>Improvement of <strong>Computational power by GPU</strong> <strong>and TPU</strong></li><li><strong>Deep Learning</strong> enables to Process <strong>Unstructured Data</strong></li></ul><p>OK, so now we can take the next step to understand how Data Science jobs are separated in real industries. Let’s figure out where to start your career depends on what you have right now and what you should have to get an ideal position for 5–10 years long-span career plan.</p><p>— — —</p><h3>2. Data Scientist vs Machine Learning Engineer</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*gD6xx6tjuFmR4WCvGm-mmQ.png" /><figcaption>from <a href="http://www.stoodnt.com/">www.stoodnt.com</a></figcaption></figure><p>In 2018 summer,<strong> I decided to build a Machine Learning related career</strong> from general Data Analyst job because I was pretty sure this technological innovation will exactly reproduce what we’ve seen <strong>a drastic change which was caused by internet and smartphone</strong>.</p><p>Suppose you already heard about the average salary of data scientist (<strong>117,000USD/year!</strong>) or understood enough the potential of ML innovation. Here, I only focus on talking to people who want to switch their expertise domain into ML(data science) by taking several years.</p><p>After I built my <a href="https://samurainote.github.io/"><strong>Data Science portfolio</strong></a> and started my job hunting I realized there are mainly two different job title which can work closely with machine learning technology:</p><blockquote>One is Data Scientist. Another is Machine Learning Engineer.</blockquote><p>As you can see above image, the difference between these two and classic <strong>Data Analyst</strong> job and <strong>Data Engineer</strong>(<strong>Big Data Engineer</strong>) job is relatively easy to describe because they’ve been already existing for more than 10 years and required skills are very clear.</p><p>However, I found the required skills and knowledge of Data Scientists and Machine Learning Engineer is quite <strong>duplicated</strong>.</p><p>The reason is simple. Since <strong>Machine Learning is the most impactful innovation in recent Data Science field</strong>, which even able to create new business and companies like Chatbot startup or Drone startup, the specialist of Machine Learning itself became an undoubtedly respectable job title.</p><p>On the other hand, nowadays we also cannot talk about Data Science without taking Machine Learning into account, Data Scientist also must know ML Theory as well. Because <strong>most of the companies are currently interested in to accelerate their business</strong> by using ML technology.</p><p>In fact, Data Scientists might not only have theoretical knowledge of Machine Learning but also, at least, be able to implement several Machine Learning Algorithm like SVM or Random Forest for classification task by using scikit-learn or build Neural Network by using Keras.</p><p>Because ironically or interestingly, unless you understand math and statistics deeply, if we want to understand ML theory and Deep learning, it’s an efficient or required way to write codes and implement it by using Tensorflow and scikit-learn or those high-level API.</p><h3>2–1. The different Area of Expertise between DS and ML engineer</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r_6C982MhL-Zu9e2m6YmcA.jpeg" /><figcaption>Related skills in DS field from <a href="https://www.edureka.co/blog/what-is-data-science/">edureka!</a></figcaption></figure><p>I know <a href="https://www.quora.com/What-is-difference-between-a-Data-Scientist-and-a-Machine-Learning-Engineer-Which-is-better-in-terms-of-salary-and-long-term-growth-and-why-How-are-they-correlated">this highest rated answer for this question from Quora</a> is not enough for you, to clarify your career in this field.</p><p>Finally, I found the wall which is unable to climb over between these two jobs in real industries. It was like this:</p><blockquote>Professional Machine Learning Engineer can build an “end-to-end software product” which has machine learning algorithm as a part of them.</blockquote><blockquote>Professional Data Scientists can define “the problem which should be solved(or not)” with scalability by using by machine learning and “How” as well.</blockquote><p>I hope you get some pictures of what I wanna say. Please don’t forget that <strong>the final goal and responsible mission of ML engineer are, I think, finalizing to build a moving software</strong>. More clearly said, unless you don’t have experience of backend software engineering, it seems hard to get a comprehensive ML engineering position which we can often find on JDs.</p><p>I wanted to tell the newbies from a non-engineering background, like data analyst, the reality is that some serious tech companies write neural network from scratch, I mean <strong>they even don’t rely on Keras or Tensorflow</strong>.</p><p>On the other hand, Data Scientist requires outstanding business understanding which is more vague and difficult to prove though (life is hard). But this is so important because in some case, a classical statistics method like Multiple regression analysis can be applied, ML is even not required. And also the application of ML in software requires an enormous amount of time and human resources. It’s necessary to calculate cost-performance balance before a huge investment decision making.</p><h3>2–2. So is it impossible to get ML engineering job?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r38evWNREo-rj1OR-24-DA.jpeg" /><figcaption>image from <a href="http://www.newtium.com/">www.newtium.com</a></figcaption></figure><p>Well, for the enthusiastic ML fresher, I found a suitable position for us in ML projects. That is <strong>Data Preprocessing and Feature Engineering </strong>role.</p><p>In the real machine learning application, we repeat the following main process again and again until we acquire significant enough accuracy:</p><ol><li><strong>Data Preprocessing and Feature Engineering</strong></li><li><strong>Modeling ML/DL architecture and Training</strong></li><li><strong>Model Validation and Hyperparameter Tuning</strong></li></ol><p>Then finally ML engineers put this architecture into existing software or define whole architecture at the same time if you build a new product. In this process, it requires more development experiences and knowledge.</p><p>— — —</p><h3>3. Required Skill for A Full-Stuck Data Scientist</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*JVW438AWfBfxuDknGrJgFA.jpeg" /><figcaption><a href="https://www.stoodnt.com/blog/data-engineer-vs-data-scientist/">What is the difference between Data Scientist and Data Engineer?</a></figcaption></figure><blockquote>Conclusion: In my definition, A Full-Stack Data Scientist is a perfect mix of Data Scientist and Machine Learning Engineer, who can design and build “End-to-End Machine Learning Project and Software”. — by me</blockquote><p>After <strong>I analyzed over 200 job description of the Machine Learning related position</strong> in Japan and India, Singapore, including companies like Google, Facebook, IBM. I found <strong>must-have skill</strong> towards A Full-Stack Data Scientist Career.</p><p>I will divide them into two different categories, one is a visible and more practical skill(<strong>more important in terms of getting a job!</strong>), another is theoretical and relatively difficult to prove.</p><p>You can use the following checklist before you start making a learning plan. It’s very flexible as well depends on JDs which you want to apply.</p><h3>3–1. Practical Skill (Easy to prove and visualize)</h3><ul><li><strong>Basic Statistical Language</strong>: Python, R, Julia</li><li><strong>Data Science Library</strong>: Numpy, Pandas, Scipy, Seaborn</li><li><strong>ML/DL Library Experience</strong>: Tensorflow, Torch, scikit-learn</li><li><strong>Unstructured Data Processing: </strong>Image, Text, Sounds</li><li><strong>Relational Database</strong>: MySQL, PostgreSQL, SQLite</li><li><strong>Distributed File System</strong>: Hadoop, Spark, AWS, MongoDB</li><li><strong>Container-type virtual environment</strong>: Docker</li><li><strong>Version control system: </strong>GitHub</li><li><strong>Web Framework</strong>: Django, Flask, Ruby on Rails</li></ul><h3>Additional Tips: to make a difference from others</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RkJs7CEZUzkTJygvMdhWJg.png" /><figcaption><a href="http://jasonkgoodman.com/">http://jasonkgoodman.com/</a></figcaption></figure><p>I strongly recommend understanding the significance of visualizing your skills. <a href="http://varianceexplained.org/r/year_data_scientist/">The serious people who already got Data Scientist Job </a>are always good at not only Data Visualization but also Skill Visualization. Please take a look at this impressive DS portfolio of <a href="http://jasonkgoodman.com/">Jason Goodman</a> as the best example.</p><h3>3–2. Theoretical Understanding (Difficult to visualize)</h3><ul><li><strong>ML Theory</strong>: Algorithm, Hyperparameters, Gradient Descent</li><li><strong>DL Theory</strong>: CNN, LSTM, Back Propagation</li><li><strong>Math</strong>: Algebra, Calculus, Probability</li><li><strong>Statistics</strong>: Bayes, Variance and Standard Deviation</li><li><strong>Computer Science</strong>: OS, Server, CPU, Data Structure, and Algorithm</li></ul><p>I think you need some trick to prove your understanding of these theories because it’s not something easy to show. Of course, the most powerful one is having the related academic degrees, next is certifications.</p><p><strong><em>Note</em></strong><em>: I highly recommend </em><strong><em>building your own publications and writing articles and show your theoretical understanding of this amazing Medium </em></strong><em>like my </em><a href="https://medium.com/shortcutnlp"><em>Coldstart.ml</em></a><em>. More Efficient way is to publish your article via </em><a href="https://towardsdatascience.com/"><em>Towards Data Science</em></a><em> or </em><a href="https://hackernoon.com/"><em>Hacker Noon</em></a><em>.</em></p><p>— — —</p><h3>4. A Golden DS Learning path for a newbie</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*hwjbI0KXMhxVIpAqqUl_LA.jpeg" /><figcaption>from <a href="https://www.datacamp.com/community/blog/full-stack-data-science-transcript">Data Camp</a></figcaption></figure><blockquote>You don’t have to master all of these tools at the first step of your career, but better to know than nothing. — by James from somewhere</blockquote><p>Since the requirement of data scientist job and ML engineer position needed <strong>a wide range of skills with a certain quality</strong>. That’s why if the market is quite competitive and you want to get closer to their ideal candidates, you can take a step by step. <strong>Organize your career plan, not in short terms!</strong> I mean you can also start from Data Base Engineer or Backend Engineer to strengthen your IT background as a first step.</p><p>For the person who is totally new to this field, like me in 9 months back, I will describe where to start your endless Data Science Learning to perform well in your real business application. It takes 12 steps and please remember you can also acquire these skill through your job. Let’s take a look:</p><ol><li><strong>Basic Statistical Language Handling</strong>: Python, R, Julia</li><li><strong>Data Science Library</strong>: Numpy, Pandas, Scipy, Seaborn</li><li><strong>Version control system: </strong>GitHub</li><li><strong>Math</strong>: Algebra, Calculus, Probability</li><li><strong>Statistics</strong>: Bayes, Variance and Standard Deviation</li><li><strong>ML/DL Library Experience</strong>: Tensorflow, Torch, scikit-learn</li><li><strong>ML Theory</strong>: Algorithm, Hyperparameters, Gradient Descent</li><li><strong>Unstructured Data Processing: </strong>Image, Text, Sounds</li><li><strong>DL Theory</strong>: CNN, LSTM, Back Propagation</li><li><strong>Relational Database</strong>: MySQL, PostgreSQL, SQLite</li><li><strong>Distributed File System</strong>: Hadoop, Spark, AWS, MongoDB</li><li><strong>Web Framework</strong>: Django, Flask, Ruby on Rails</li></ol><p><strong>+Job Hunting Technics</strong>: networking, resume, interview</p><p>As a note, I applied some principle in the above order to focus on getting a job:</p><ul><li><strong>Build your own Data Science Portfolio, not only study on MOOC</strong></li><li><strong>Spend 90% of your time to create visible outputs</strong></li><li><strong>Understanding theory through coding, so code first</strong></li></ul><p>That’s why the coding part is coming before theory, through this I want to tell you that Cousera or Udemy doesn’t help enough when you are applying for a real job. Smart people do something different. It’s good to remember!</p><p>— — —</p><h3>5. Secret Bibles towards A Full-Stack Data Scientist</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*07KkVtszeVrEkFUsL-Xv8Q.jpeg" /><figcaption><a href="https://towardsdatascience.com/understand-these-5-basic-concepts-to-sound-like-a-machine-learning-expert-6221ec0fe960">Understand these 5 basic concepts to sound like a machine learning expert</a></figcaption></figure><blockquote>I spent or even wasted a ton of time to reach out the best resources with clean code example about ML/DL. — by Me</blockquote><p>It could be a crucial problem when you want to find the learning resources which is really suitable for you. Because there are a ton of learning materials on Google and easy to be drawn in an ocean of information.</p><p>In this chapter, I’m not just going to introduce MOOC courses. Because it’s already done by perfectly. If you want to know that, I recommend knowing A leading person of this learning resource discussion for ML. His name is <a href="https://medium.freecodecamp.org/@davidventuri"><strong>David Venturi</strong></a><strong> </strong>who built <strong>a completely hand-made Machine Learning Master Degree program </strong>by only free learning resource on the internet. Check it out this his post!</p><p><a href="https://davidventuri.medium.com/i-dropped-out-of-school-to-create-my-own-data-science-master-s-here-s-my-curriculum-1b400dcee412">I Dropped Out of School to Create My Own Data Science Master’s — Here’s My Curriculum</a></p><p>Apart from these famous courses, I wanna summarize additional secret resources a lot of learners worship and I also highly appreciates these. I will introduce the following uniques:</p><ol><li><strong>Tutorial web site with the simplest codes</strong></li><li><strong>Online Class Session from Harvard and Stanford</strong></li><li><strong>Online PDF: which you are going to miss</strong></li><li><strong>Top 10 Insightful article from Medium</strong></li></ol><p>Now please prepare your Evernote and copy them with the title of <strong>“Bibles of Data Science”</strong>. It’s masterpieces from DS enthusiasts. <strong>Don’t worry it’s all free!</strong> (I know everyone loves free lunch…)<strong> </strong>Let’s take a look.</p><h3>1. Top 3 Tutorial web site with the simplest codes</h3><ul><li><a href="https://machinelearningmastery.com/start-here/">Start Here with Machine Learning - Machine Learning Mastery</a></li><li><a href="https://elitedatascience.com/start-here">Start Here - EliteDataScience Orientation Page</a></li><li><a href="https://www.pyimagesearch.com/start-here-learn-computer-vision-opencv/">Start here: Learn computer vision &amp; OpenCV - PyImageSearch</a></li></ul><ul><li><a href="https://scikit-learn.org/stable/index.html">Scikit-learn official Tutorials</a></li><li><a href="https://blog.keras.io/category/tutorials.html">Keras official Tutorials</a></li><li><a href="http://deeplearning.net/tutorial/">Deep Learning Tutorials with Theano by deeplearning.net</a></li></ul><p><em>Note: About ML/DL, there is no shortcut or simplest tutorial more than scikit-learn and Keras. So don’t waste your time(I did) to find any other resources, start from their official tutorials, you’ll get strong fundamentals.</em></p><h3>2. Online Class from Harvard and Stanford</h3><p><a href="http://cs109.github.io/2015/">CS109 Data Science</a></p><ul><li><a href="https://www.youtube.com/playlist?list=PLf7L7Kg8_FNxHATtLwDceyh72QQL9pvpQ"><strong>Computer Vision Lecture Video from Stanford University</strong></a></li><li><a href="https://www.youtube.com/watch?v=OQQ-W_63UgQ"><strong>Natural Language Processing Lecture Video from Stanford University</strong></a></li></ul><h3>3. Online PDF: which you are going to miss</h3><h4>Highly Recommend:</h4><ul><li><a href="http://www.ic.unicamp.br/~wainer/cursos/1s2013/ml/livro.pdf">All of Statistics: A Concise Course in Statistical Inference</a></li><li><a href="http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf">Pattern Recognition and Machine Learning</a></li><li><a href="http://faculty.neu.edu.cn/yury/AAI/Textbook/Deep%20Learning%20with%20Python.pdf">Deep Learning with Python</a></li></ul><h4>Other Specials</h4><ul><li><a href="http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/091117.pdf">Bayesian Reasoning and Machine Learning</a></li><li><a href="http://www-bcf.usc.edu/~gareth/ISL/">An Introduction to Statistical Learning</a></li><li><a href="https://web.stanford.edu/~hastie/ElemStatLearn//printings/ESLII_print12.pdf">The Elements of Statistical Learning Data Mining, Inference, and Prediction</a></li><li><a href="http://www.inference.org.uk/itprnn/book.pdf">Information Theory, Inference, and Learning Algorithms</a></li><li><a href="https://www.deeplearningbook.org/front_matter.pdf">Deep Learning Ian Goodfellow</a></li><li><a href="http://szeliski.org/Book/drafts/SzeliskiBook_20100903_draft.pdf">Computer Vision: Algorithms and Applications</a></li><li><a href="http://ling.snu.ac.kr/class/AI_Agent/deep_learning_for_nlp.pdf">Deep Learning for Natural Language Processing</a></li><li><a href="http://www.datascienceassn.org/sites/default/files/Natural%20Language%20Processing%20with%20Python.pdf">Natural Language Processing with Python</a></li><li><a href="https://web.stanford.edu/~jurafsky/slp3/ed3book.pdf">Speech and Language Processing</a></li><li><a href="http://greenteapress.com/thinkstats/thinkstats.pdf">Think Stats: Probability and Statistics for Programmers</a></li></ul><h3>4. Top 10 Insightful article from Medium</h3><ul><li><a href="https://elitedatascience.com/learn-machine-learning">How to Learn Machine Learning, The Self-Starter Way</a></li><li><a href="https://hackernoon.com/what-on-earth-is-data-science-eb1237d8cb37">What on earth is data science?</a></li><li><a href="https://medium.com/machine-learning-for-humans/why-machine-learning-matters-6164faf1df12">A Beginner’s Guide to AI/ML 🤖👶</a></li><li><a href="https://hackernoon.com/why-businesses-fail-at-machine-learning-fbff41c4d5db">Why businesses fail at machine learning</a></li><li><a href="https://medium.com/deliberate-data-science/12-things-i-wish-id-known-before-starting-as-a-data-scientist-45989be6300e">12 things I wish I’d known before starting as a Data Scientist</a></li><li><a href="https://towardsdatascience.com/being-a-data-scientist-does-not-make-you-a-software-engineer-c64081526372">Being a Data Scientist does not make you a Software Engineer!</a></li><li><a href="https://towardsdatascience.com/essential-math-for-data-science-why-and-how-e88271367fbd">Essential Math for Data Science</a></li><li><a href="https://medium.com/data-deft/probability-statistics-for-data-science-series-83b94353ca48">Probability &amp; Statistics for Data Science (Series)</a></li><li><a href="https://towardsdatascience.com/linear-algebra-for-deep-learning-506c19c0d6fa">Linear Algebra for Deep Learning</a></li><li><a href="https://towardsdatascience.com/why-data-scientists-love-gaussian-6e7a7b726859">Why Data Scientists love Gaussian?</a></li></ul><p>— — —</p><h3>Conclusion</h3><blockquote>“Better Data &gt; Fancier Algorithms“ — <a href="https://elitedatascience.com/">EliteDataScience.com</a></blockquote><p>Since this new field is changing so fast, nobody knows which skills will be commoditized or keep their values longer than others. In reality, the Data Science consulting company who I met on my job hunting process said, “we just integrated our company with Ad platforms because making difference only by analytical quality or knowledge about Data Science became so hard. We need our own unique big data.”</p><p>But even in this kinda situation, we need to constantly update our input information and predict what will happen in the future of one step ahead. We have no time to waste, we have to bet or invest time and learning in a precise direction. Let’s see what will happen.</p><p>— — —</p><h3>Takeaway: 10 Recommendations to encourage you for your ML careers!</h3><ul><li><a href="https://towardsdatascience.com/how-to-land-a-data-scientist-job-at-your-dream-company-my-journey-to-airbnb-f6a1e99892e8">How to land a Data Scientist job at your dream company - My journey to Airbnb</a></li><li><a href="https://medium.com/@XiaohanZeng/i-interviewed-at-five-top-companies-in-silicon-valley-in-five-days-and-luckily-got-five-job-offers-25178cf74e0f">I interviewed at five top companies in Silicon Valley in five days, and luckily got five job offers</a></li><li><a href="https://blog.usejournal.com/how-i-got-7-job-offers-in-8-weeks-part-1-please-interview-me-21e6f4ded106">How I got 7 Job Offers in 8 Weeks (Part 1: Please Interview Me?)</a></li><li><a href="https://medium.com/darius-foroux/how-to-get-your-dream-job-without-experience-73627048e856">How To Get Your Dream Job Without Experience</a></li><li><a href="https://medium.com/@datalab/how-to-break-into-the-data-science-market-f0e0b79b42f7">How to get your first job in Data Science?</a></li><li><a href="https://towardsdatascience.com/six-recommendations-for-aspiring-data-scientists-93d12aeb9b9">Six Recommendations for Aspiring Data Scientists</a></li><li><a href="https://towardsdatascience.com/how-to-build-a-data-science-portfolio-5f566517c79c">How to Build a Data Science Portfolio</a></li><li><a href="https://medium.com/@jasonkgoodman/advice-on-building-data-portfolio-projects-c5f96d8a0627">Advice on Building Data Portfolio Projects</a></li><li><a href="https://towardsdatascience.com/the-cold-start-problem-how-to-build-your-machine-learning-portfolio-6718b4ae83e9">The cold start problem: how to build your machine learning portfolio</a></li><li><a href="https://medium.com/machine-learning-in-practice/cheat-sheet-of-machine-learning-and-python-and-math-cheat-sheets-a4afe4e791b6">Cheat Sheet of Machine Learning and Python (and Math) Cheat Sheets</a></li></ul><p>See you in the Next Article! ありがとう！</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e865e47e3788" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Golden path to become A Full-stuck Data Scientist who can work for the real industry task from…]]></title>
            <link>https://medium.com/shortcutnlp/the-golden-path-to-become-a-full-stuck-data-scientist-who-can-work-for-the-real-industry-task-from-22dade17dfd?source=rss-bc2291698c8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/22dade17dfd</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Akira Takezawa]]></dc:creator>
            <pubDate>Mon, 25 Mar 2019 12:32:49 GMT</pubDate>
            <atom:updated>2019-03-27T03:29:28.312Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Golden Path to become A Full-Stack Data Scientist.</h3><h4>Actual report from 180 days of job shifting to ML field</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LUBN4_IVEO2pkoAJ2qAzPg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/kdNs6NZMaD4?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Falco Negenman</a> on <a href="https://unsplash.com/search/photos/art?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Welcome to <a href="https://medium.com/shortcutnlp">Coldstart.ml</a> which provides variable Data Science information for enthusiastic learners!</p><h3>Why do you have to read this?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*PuDcUlhu6ccUV3p493CJ2Q.png" /><figcaption><a href="https://www.glassdoor.com/Salaries/data-scientist-salary-SRCH_KO0,14.htm">from glassdoor.com on 25th of March, 2019</a></figcaption></figure><blockquote>The shortage of data scientists is becoming a serious constraint in some sectors. — <a href="https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century">Data Scientist: The Sexiest Job of the 21st Century</a> from Havard Business Review</blockquote><p>Nowadays <strong>Big Data</strong> is one of the most important key resources to get a competitive advantage in business, especially for IT companies. All of <a href="https://www.quora.com/What-is-GAFA">GAFA</a>, accelerate their business by applying Data Science Technique. Let me explain a little bit more.</p><p>For instance, Google, <strong>Youtube uses Recommend Engine</strong> which will never let you go, because they suggest contents by completely following your taste. <strong>Amazon increases its </strong><a href="https://rnnnews.jp/search/result/?q=gross+merchandise+volume"><strong>Gross Merchandise Volume</strong></a>(GMS) by matching user and product very efficiently. <strong>Facebook shows Advertisements with higher </strong><a href="https://www.mobvista.com/en/blog/what-are-cvr-and-ctr/"><strong>CVR</strong></a> from extracting critical insights from your demographical and behavioral data. (I used their Ads in the previous job and was surprised at how effective it performed.)</p><p>Till now I’ve only mentioned about Internet Giants. However, according to <a href="https://money.cnn.com/interactive/technology/masayoshi-son-profile/index.html"><strong>Masayoshi Son</strong></a> who is Japanese CEO of SoftBank Vision Fund’s which is one of the biggest founds all over the world, <a href="https://asia.nikkei.com/Business/Companies/SoftBank-Vision-Fund-s-investments-hover-near-45bn-in-2018">and investments hover near $45 Billion dollars in 2018</a>, left an insightful comment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GLJGZ3L-U2HVN92fS1LcWg.jpeg" /><figcaption>image from SoftBank’s unicorn hunter</figcaption></figure><blockquote>The impact of Internet is somehow limited in particular domains or industries, like Advertisement, Retail (E-commerce), but AI is different. — Masoyoshi Son</blockquote><p><strong>“Next Big Waves”</strong> in other industries, which is applying data science methods into <strong>Unstructured Data </strong>like Image, Natural Language, Sounds, are coming up. I mean, for example, <strong>Mobility Industry </strong>you can see what Elon Mask does, <strong>Robotics</strong> you see what’s happening in Amazon’s warehouse, <strong>Finance</strong>, <strong>Healthcare</strong> area and so on.</p><p><strong>If you are young and wanna be the biggest winner in your career</strong>, this Data Science or Machine Learning field could be the most possible choice.</p><p>Because from my marketing perspective, a quite simple demand-supply balance analysis, I think some special paid job title like professionals in the Finance field or classic Software Engineer, Business Development position with MBA grad is<strong> already taken by our elders</strong>. And those skills and knowledge are became commoditized. I mean these are very competitive.</p><p>Comparing to them, <strong>Data Science field is way messier and still in the mist.</strong> Then why not invest your career and passion, intelligence into this challenging field? Welcome to this fantastic ML technology with a full of hope!</p><p>In this article, I will focus on h<strong>ow to get a Data Scientist jobs or Machine Learning Engineer posts </strong>who will be needed by real industries, or in other words, who will pass the job hunting process with lesser pain. Let’s get started!</p><p>— — —</p><h3>After you read this, you’ll get:</h3><ul><li><strong>The Shortest Path</strong> towards A Real Data Scientist</li><li><strong>The Best Learning Resource</strong> everyone trust in this area</li><li><strong>The Realistic Possibility</strong> of your career with ML</li></ul><p>— — —</p><h3>Menu</h3><ol><li>What is Data Science for Business?</li><li>Data scientist vs Machine Learning Engineer</li><li>Required Skill for A Full-Stuck Data Scientist</li><li>A Golden DS Learning path for a newbie</li><li>The Secret Bibles towards A Full-Stack Data Scientist</li></ol><p>— — —</p><h3>1. What is Data Science for Business?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SyaztIXrRAPj2vADJPICEA.png" /><figcaption>from <a href="https://www.edureka.co/blog/what-is-machine-learning/">edureka!</a></figcaption></figure><blockquote>Data Science itself is just a Method, Not a Goal in our business. Then definition of it should be explained in much simpler way. — an unknown data scientists</blockquote><p>Data Science itself can never be <strong>a purpose</strong>, if it becomes, it means you already fail. Rather than that, I would say “We just became able to make some <strong>additional or novel values</strong> in the real business which we couldn’t 10 years before”.</p><p>I highly recommend listening to this podcast, <a href="https://www.superdatascience.com/podcast/podcast-one-purpose-data-science-truth-analytics"><strong>SDS 131: The One Purpose to Data Science and The Truth about Analytics</strong></a> from <a href="https://www.superdatascience.com/">SuperDataScience</a>.</p><p>To define Data Science, I wanted to start from their goal and purpose. <strong>The fundamental goals of Data Science</strong> in business is pretty clear. I think it can be simply described as followings:</p><ul><li>To <strong>make more profit</strong> in your business by using data (as a result)</li><li>To understand and <strong>satisfy your customers</strong> by efficiency, better matching</li><li>To <strong>create a new business</strong> or startup by using machine learning</li></ul><h3><strong>The background behind this Game Change</strong></h3><p>In addition, I think it’s very important to understand the reasons why the significance of Data Science is gradually recognized in recent business. It could be the following three:</p><ul><li>An explosion the <strong>Amount of Data by Internet and Smartphone</strong></li><li>Improvement of <strong>Computational power by GPU</strong> <strong>and TPU</strong></li><li><strong>Deep Learning</strong> enables to Process <strong>Unstructured Data</strong></li></ul><p>OK, so now we can take the next step to understand how Data Science jobs are separated in real industries. Let’s figure out where to start your career depends on what you have right now and what you should have to get an ideal position for 5–10 years long-span career plan.</p><p>— — —</p><h3>2. Data Scientist vs Machine Learning Engineer</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*gD6xx6tjuFmR4WCvGm-mmQ.png" /><figcaption>from <a href="http://www.stoodnt.com/">www.stoodnt.com</a></figcaption></figure><p>In 2018 summer,<strong> I decided to build a Machine Learning related career</strong> from general Data Analyst job because I was pretty sure this technological innovation will exactly reproduce what we’ve seen <strong>a drastic change which was caused by internet and smartphone</strong>.</p><p>Suppose you already heard about the average salary of data scientist (<strong>117,000USD/year!</strong>) or understood enough the potential of ML innovation. Here, I only focus on talking to people who want to switch their expertise domain into ML(data science) by taking several years.</p><p>After I built my <a href="https://samurainote.github.io/"><strong>Data Science portfolio</strong></a> and started my job hunting I realized there are mainly two different job title which can work closely with machine learning technology:</p><blockquote>One is Data Scientist. Another is Machine Learning Engineer.</blockquote><p>As you can see above image, the difference between these two and classic <strong>Data Analyst</strong> job and <strong>Data Engineer</strong>(<strong>Big Data Engineer</strong>) job is relatively easy to describe because they’ve been already existing for more than 10 years and required skills are very clear.</p><p>However, I found the required skills and knowledge of Data Scientists and Machine Learning Engineer is quite <strong>duplicated</strong>.</p><p>The reason is simple. Since <strong>Machine Learning is the most impactful innovation in recent Data Science field</strong>, which even able to create new business and companies like Chatbot startup or Drone startup, the specialist of Machine Learning itself became an undoubtedly respectable job title.</p><p>On the other hand, nowadays we also cannot talk about Data Science without taking Machine Learning into account, Data Scientist also must know ML Theory as well. Because <strong>most of the companies are currently interested in to accelerate their business</strong> by using ML technology.</p><p>In fact, Data Scientists might not only have theoretical knowledge of Machine Learning but also, at least, be able to implement several Machine Learning Algorithm like SVM or Random Forest for classification task by using scikit-learn or build Neural Network by using Keras.</p><p>Because ironically or interestingly, unless you understand math and statistics deeply, if we want to understand ML theory and Deep learning, it’s an efficient or required way to write codes and implement it by using Tensorflow and scikit-learn or those high-level API.</p><h3><strong>2–1. The different Area of Expertise between DS and ML engineer</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r_6C982MhL-Zu9e2m6YmcA.jpeg" /><figcaption>Related skills in DS field from <a href="https://www.edureka.co/blog/what-is-data-science/">edureka!</a></figcaption></figure><p>I know <a href="https://www.quora.com/What-is-difference-between-a-Data-Scientist-and-a-Machine-Learning-Engineer-Which-is-better-in-terms-of-salary-and-long-term-growth-and-why-How-are-they-correlated">this highest rated answer for this question from Quora</a> is not enough for you, to clarify your career in this field.</p><p>Finally, I found the wall which is unable to climb over between these two jobs in real industries. It was like this:</p><blockquote>Professional Machine Learning Engineer can build an “end-to-end software product” which has machine learning algorithm as a part of them.</blockquote><blockquote>Professional Data Scientists can define “the problem which should be solved(or not)” with scalability by using by machine learning and “How” as well.</blockquote><p>I hope you get some pictures of what I wanna say. Please don’t forget that <strong>the final goal and responsible mission of ML engineer are, I think, finalizing to build a moving software</strong>. More clearly said, unless you don’t have experience of backend software engineering, it seems hard to get a comprehensive ML engineering position which we can often find on JDs.</p><p>I wanted to tell the newbies from a non-engineering background, like data analyst, the reality is that some serious tech companies write neural network from scratch, I mean <strong>they even don’t rely on Keras or Tensorflow</strong>.</p><p>On the other hand, Data Scientist requires outstanding business understanding which is more vague and difficult to prove though (life is hard). But this is so important because in some case, a classical statistics method like Multiple regression analysis can be applied, ML is even not required. And also the application of ML in software requires an enormous amount of time and human resources. It’s necessary to calculate cost-performance balance before a huge investment decision making.</p><h3><strong>2–2. So is it impossible to get ML engineering job?</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r38evWNREo-rj1OR-24-DA.jpeg" /><figcaption>image from <a href="http://www.newtium.com/">www.newtium.com</a></figcaption></figure><p>Well, for the enthusiastic ML fresher, I found a suitable position for us in ML projects. That is <strong>Data Preprocessing and Feature Engineering </strong>role.</p><p>In the real machine learning application, we repeat the following main process again and again until we acquire significant enough accuracy:</p><ol><li><strong>Data Preprocessing and Feature Engineering</strong></li><li><strong>Modeling ML/DL architecture and Training</strong></li><li><strong>Model Validation and Hyperparameter Tuning</strong></li></ol><p>Then finally ML engineers put this architecture into existing software or define whole architecture at the same time if you build a new product. In this process, it requires more development experiences and knowledge.</p><p>— — —</p><h3>3. Required Skill for A Full-Stuck Data Scientist</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*JVW438AWfBfxuDknGrJgFA.jpeg" /><figcaption><a href="https://www.stoodnt.com/blog/data-engineer-vs-data-scientist/">What is the difference between Data Scientist and Data Engineer?</a></figcaption></figure><blockquote>Conclusion: In my definition, A Full-Stack Data Scientist is a perfect mix of Data Scientist and Machine Learning Engineer, who can design and build “End-to-End Machine Learning Project and Software”. — by me</blockquote><p>After <strong>I analyzed over 200 job description of the Machine Learning related position</strong> in Japan and India, Singapore, including companies like Google, Facebook, IBM. I found <strong>must-have skill</strong> towards A Full-Stack Data Scientist Career.</p><p>I will divide them into two different categories, one is a visible and more practical skill(<strong>more important in terms of getting a job!</strong>), another is theoretical and relatively difficult to prove.</p><p>You can use the following checklist before you start making a learning plan. It’s very flexible as well depends on JDs which you want to apply.</p><h3><strong>3-1. Practical Skill (Easy to prove and visualize)</strong></h3><ul><li><strong>Basic Statistical Language</strong>: Python, R, Julia</li><li><strong>Data Science Library</strong>: Numpy, Pandas, Scipy, Seaborn</li><li><strong>ML/DL Library Experience</strong>: Tensorflow, Torch, scikit-learn</li><li><strong>Unstructured Data Processing: </strong>Image, Text, Sounds</li><li><strong>Relational Database</strong>: MySQL, PostgreSQL, SQLite</li><li><strong>Distributed File System</strong>: Hadoop, Spark, AWS, MongoDB</li><li><strong>Container-type virtual environment</strong>: Docker</li><li><strong>Version control system: </strong>GitHub</li><li><strong>Web Framework</strong>: Django, Flask, Ruby on Rails</li></ul><h3><strong>Additional Tips: to make a difference from others</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RkJs7CEZUzkTJygvMdhWJg.png" /><figcaption><a href="http://jasonkgoodman.com/">http://jasonkgoodman.com/</a></figcaption></figure><p>I strongly recommend understanding the significance of visualizing your skills. <a href="http://varianceexplained.org/r/year_data_scientist/">The serious people who already got Data Scientist Job </a>are always good at not only Data Visualization but also Skill Visualization. Please take a look at this impressive DS portfolio of <a href="http://jasonkgoodman.com/">Jason Goodman</a> as the best example.</p><h3><strong>3-2. Theoretical Understanding (Difficult to visualize)</strong></h3><ul><li><strong>ML Theory</strong>: Algorithm, Hyperparameters, Gradient Descent</li><li><strong>DL Theory</strong>: CNN, LSTM, Back Propagation</li><li><strong>Math</strong>: Algebra, Calculus, Probability</li><li><strong>Statistics</strong>: Bayes, Variance and Standard Deviation</li><li><strong>Computer Science</strong>: OS, Server, CPU, Data Structure, and Algorithm</li></ul><p>I think you need some trick to prove your understanding of these theories because it’s not something easy to show. Of course, the most powerful one is having the related academic degrees, next is certifications.</p><p><strong><em>Note</em></strong><em>: I highly recommend </em><strong><em>building your own publications and writing articles and show your theoretical understanding of this amazing Medium </em></strong><em>like my </em><a href="https://medium.com/shortcutnlp"><em>Coldstart.ml</em></a><em>. More Efficient way is to publish your article via </em><a href="https://towardsdatascience.com/"><em>Towards Data Science</em></a><em> or </em><a href="https://hackernoon.com/"><em>Hacker Noon</em></a><em>.</em></p><p>— — —</p><h3>4. A Golden DS Learning path for a newbie</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*hwjbI0KXMhxVIpAqqUl_LA.jpeg" /><figcaption>from <a href="https://www.datacamp.com/community/blog/full-stack-data-science-transcript">Data Camp</a></figcaption></figure><blockquote>You don’t have to master all of these tools at the first step of your career, but better to know than nothing. — by James from somewhere</blockquote><p>Since the requirement of data scientist job and ML engineer position needed <strong>a wide range of skills with a certain quality</strong>. That’s why if the market is quite competitive and you want to get closer to their ideal candidates, you can take a step by step. <strong>Organize your career plan, not in short terms!</strong> I mean you can also start from Data Base Engineer or Backend Engineer to strengthen your IT background as a first step.</p><p>For the person who is totally new to this field, like me in 9 months back, I will describe where to start your endless Data Science Learning to perform well in your real business application. It takes 12 steps and please remember you can also acquire these skill through your job. Let’s take a look:</p><ol><li><strong>Basic Statistical Language Handling</strong>: Python, R, Julia</li><li><strong>Data Science Library</strong>: Numpy, Pandas, Scipy, Seaborn</li><li><strong>Version control system: </strong>GitHub</li><li><strong>Math</strong>: Algebra, Calculus, Probability</li><li><strong>Statistics</strong>: Bayes, Variance and Standard Deviation</li><li><strong>ML/DL Library Experience</strong>: Tensorflow, Torch, scikit-learn</li><li><strong>ML Theory</strong>: Algorithm, Hyperparameters, Gradient Descent</li><li><strong>Unstructured Data Processing: </strong>Image, Text, Sounds</li><li><strong>DL Theory</strong>: CNN, LSTM, Back Propagation</li><li><strong>Relational Database</strong>: MySQL, PostgreSQL, SQLite</li><li><strong>Distributed File System</strong>: Hadoop, Spark, AWS, MongoDB</li><li><strong>Web Framework</strong>: Django, Flask, Ruby on Rails</li></ol><p><strong>+Job Hunting Technics</strong>: networking, resume, interview</p><p>As a note, I applied some principle in the above order to focus on getting a job:</p><ul><li><strong>Build your own Data Science Portfolio, not only study on MOOC</strong></li><li><strong>Spend 90% of your time to create visible outputs</strong></li><li><strong>Understanding theory through coding, so code first</strong></li></ul><p>That’s why the coding part is coming before theory, through this I want to tell you that Cousera or Udemy doesn’t help enough when you are applying for a real job. Smart people do something different. It’s good to remember!</p><p>— — —</p><h3>5. Secret Bibles towards A Full-Stack Data Scientist</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*07KkVtszeVrEkFUsL-Xv8Q.jpeg" /><figcaption><a href="https://towardsdatascience.com/understand-these-5-basic-concepts-to-sound-like-a-machine-learning-expert-6221ec0fe960">Understand these 5 basic concepts to sound like a machine learning expert</a></figcaption></figure><blockquote>I spent or even wasted a ton of time to reach out the best resources with clean code example about ML/DL. — by Me</blockquote><p>It could be a crucial problem when you want to find the learning resources which is really suitable for you. Because there are a ton of learning materials on Google and easy to be drawn in an ocean of information.</p><p>In this chapter, I’m not just going to introduce MOOC courses. Because it’s already done by perfectly. If you want to know that, I recommend knowing A leading person of this learning resource discussion for ML. His name is <a href="https://medium.freecodecamp.org/@davidventuri"><strong>David Venturi</strong></a><strong> </strong>who built <strong>a completely hand-made Machine Learning Master Degree program </strong>by only free learning resource on the internet. Check it out this his post!</p><p><a href="https://davidventuri.medium.com/i-dropped-out-of-school-to-create-my-own-data-science-master-s-here-s-my-curriculum-1b400dcee412">I Dropped Out of School to Create My Own Data Science Master’s — Here’s My Curriculum</a></p><p>Apart from these famous courses, I wanna summarize additional secret resources a lot of learners worship and I also highly appreciates these. I will introduce the following uniques:</p><ol><li><strong>Tutorial web site with the simplest codes</strong></li><li><strong>Online Class Session from Harvard and Stanford</strong></li><li><strong>Online PDF: which you are going to miss</strong></li><li><strong>Top 10 Insightful article from Medium</strong></li></ol><p>Now please prepare your Evernote and copy them with the title of <strong>“Bibles of Data Science”</strong>. It’s masterpieces from DS enthusiasts. <strong>Don’t worry it’s all free!</strong> (I know everyone loves free lunch...)<strong> </strong>Let’s take a look.</p><h3><strong>1. Top 3 Tutorial web site with the simplest codes</strong></h3><ul><li><a href="https://machinelearningmastery.com/start-here/">Start Here with Machine Learning - Machine Learning Mastery</a></li><li><a href="https://elitedatascience.com/start-here">Start Here - EliteDataScience Orientation Page</a></li><li><a href="https://www.pyimagesearch.com/start-here-learn-computer-vision-opencv/">Start here: Learn computer vision &amp; OpenCV - PyImageSearch</a></li></ul><ul><li><a href="https://scikit-learn.org/stable/index.html">Scikit-learn official Tutorials</a></li><li><a href="https://blog.keras.io/category/tutorials.html">Keras official Tutorials</a></li><li><a href="http://deeplearning.net/tutorial/">Deep Learning Tutorials with Theano by deeplearning.net</a></li></ul><p><em>Note: About ML/DL, there is no shortcut or simplest tutorial more than scikit-learn and Keras. So don’t waste your time(I did) to find any other resources, start from their official tutorials, you’ll get strong fundamentals.</em></p><h3><strong>2. Online Class from Harvard and Stanford</strong></h3><p><a href="http://cs109.github.io/2015/">CS109 Data Science</a></p><ul><li><a href="https://www.youtube.com/playlist?list=PLf7L7Kg8_FNxHATtLwDceyh72QQL9pvpQ"><strong>Computer Vision Lecture Video from Stanford University</strong></a></li><li><a href="https://www.youtube.com/watch?v=OQQ-W_63UgQ"><strong>Natural Language Processing Lecture Video from Stanford University</strong></a></li></ul><h3><strong>3. Online PDF: which you are going to miss</strong></h3><h4>Highly Recommend:</h4><ul><li><a href="http://www.ic.unicamp.br/~wainer/cursos/1s2013/ml/livro.pdf">All of Statistics: A Concise Course in Statistical Inference</a></li><li><a href="http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf">Pattern Recognition and Machine Learning</a></li><li><a href="http://faculty.neu.edu.cn/yury/AAI/Textbook/Deep%20Learning%20with%20Python.pdf">Deep Learning with Python</a></li></ul><h4>Other Specials</h4><ul><li><a href="http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/091117.pdf">Bayesian Reasoning and Machine Learning</a></li><li><a href="http://www-bcf.usc.edu/~gareth/ISL/">An Introduction to Statistical Learning</a></li><li><a href="https://web.stanford.edu/~hastie/ElemStatLearn//printings/ESLII_print12.pdf">The Elements of Statistical Learning Data Mining, Inference, and Prediction</a></li><li><a href="http://www.inference.org.uk/itprnn/book.pdf">Information Theory, Inference, and Learning Algorithms</a></li><li><a href="https://www.deeplearningbook.org/front_matter.pdf">Deep Learning Ian Goodfellow</a></li><li><a href="http://szeliski.org/Book/drafts/SzeliskiBook_20100903_draft.pdf">Computer Vision: Algorithms and Applications</a></li><li><a href="http://ling.snu.ac.kr/class/AI_Agent/deep_learning_for_nlp.pdf">Deep Learning for Natural Language Processing</a></li><li><a href="http://www.datascienceassn.org/sites/default/files/Natural%20Language%20Processing%20with%20Python.pdf">Natural Language Processing with Python</a></li><li><a href="https://web.stanford.edu/~jurafsky/slp3/ed3book.pdf">Speech and Language Processing</a></li><li><a href="http://greenteapress.com/thinkstats/thinkstats.pdf">Think Stats: Probability and Statistics for Programmers</a></li></ul><h3><strong>4. Top 10 Insightful article from Medium</strong></h3><ul><li><a href="https://elitedatascience.com/learn-machine-learning">How to Learn Machine Learning, The Self-Starter Way</a></li><li><a href="https://hackernoon.com/what-on-earth-is-data-science-eb1237d8cb37">What on earth is data science?</a></li><li><a href="https://medium.com/machine-learning-for-humans/why-machine-learning-matters-6164faf1df12">A Beginner’s Guide to AI/ML 🤖👶</a></li><li><a href="https://hackernoon.com/why-businesses-fail-at-machine-learning-fbff41c4d5db">Why businesses fail at machine learning</a></li><li><a href="https://medium.com/deliberate-data-science/12-things-i-wish-id-known-before-starting-as-a-data-scientist-45989be6300e">12 things I wish I’d known before starting as a Data Scientist</a></li><li><a href="https://towardsdatascience.com/being-a-data-scientist-does-not-make-you-a-software-engineer-c64081526372">Being a Data Scientist does not make you a Software Engineer!</a></li><li><a href="https://towardsdatascience.com/essential-math-for-data-science-why-and-how-e88271367fbd">Essential Math for Data Science</a></li><li><a href="https://medium.com/data-deft/probability-statistics-for-data-science-series-83b94353ca48">Probability &amp; Statistics for Data Science (Series)</a></li><li><a href="https://towardsdatascience.com/linear-algebra-for-deep-learning-506c19c0d6fa">Linear Algebra for Deep Learning</a></li><li><a href="https://towardsdatascience.com/why-data-scientists-love-gaussian-6e7a7b726859">Why Data Scientists love Gaussian?</a></li></ul><p>— — —</p><h3>Conclusion</h3><blockquote>“Better Data &gt; Fancier Algorithms“ — <a href="https://elitedatascience.com/">EliteDataScience.com</a></blockquote><p>Since this new field is changing, nobody knows which skills will be commoditized or keep their values longer than others. In reality, the Data Science consulting company who I met on my job hunting process said, “we are going to sell our company</p><p>so fast, i. But we need to constantly update our</p><p>— — —</p><h3>Thank you for reading!</h3><blockquote>I’m still searching for my first job in Machine Learning related filed! If you enjoyed in my article, please visit my <a href="https://samurainote.github.io/">portfolio website</a>, and contact me!</blockquote><p>My name is Akira, 25y.o., male, from Tokyo, Japan, with endless curiosity.</p><p>— — —</p><h3>10 Recommendations to encourage you for your ML careers!</h3><ul><li><a href="https://towardsdatascience.com/how-to-land-a-data-scientist-job-at-your-dream-company-my-journey-to-airbnb-f6a1e99892e8">How to land a Data Scientist job at your dream company - My journey to Airbnb</a></li><li><a href="https://medium.com/@XiaohanZeng/i-interviewed-at-five-top-companies-in-silicon-valley-in-five-days-and-luckily-got-five-job-offers-25178cf74e0f">I interviewed at five top companies in Silicon Valley in five days, and luckily got five job offers</a></li><li><a href="https://blog.usejournal.com/how-i-got-7-job-offers-in-8-weeks-part-1-please-interview-me-21e6f4ded106">How I got 7 Job Offers in 8 Weeks (Part 1: Please Interview Me?)</a></li><li><a href="https://medium.com/darius-foroux/how-to-get-your-dream-job-without-experience-73627048e856">How To Get Your Dream Job Without Experience</a></li><li><a href="https://medium.com/@datalab/how-to-break-into-the-data-science-market-f0e0b79b42f7">How to get your first job in Data Science?</a></li><li><a href="https://towardsdatascience.com/six-recommendations-for-aspiring-data-scientists-93d12aeb9b9">Six Recommendations for Aspiring Data Scientists</a></li><li><a href="https://towardsdatascience.com/how-to-build-a-data-science-portfolio-5f566517c79c">How to Build a Data Science Portfolio</a></li><li><a href="https://medium.com/@jasonkgoodman/advice-on-building-data-portfolio-projects-c5f96d8a0627">Advice on Building Data Portfolio Projects</a></li><li><a href="https://towardsdatascience.com/the-cold-start-problem-how-to-build-your-machine-learning-portfolio-6718b4ae83e9">The cold start problem: how to build your machine learning portfolio</a></li><li><a href="https://medium.com/machine-learning-in-practice/cheat-sheet-of-machine-learning-and-python-and-math-cheat-sheets-a4afe4e791b6">Cheat Sheet of Machine Learning and Python (and Math) Cheat Sheets</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22dade17dfd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shortcutnlp/the-golden-path-to-become-a-full-stuck-data-scientist-who-can-work-for-the-real-industry-task-from-22dade17dfd">The Golden path to become A Full-stuck Data Scientist who can work for the real industry task from…</a> was originally published in <a href="https://medium.com/shortcutnlp">Coldstart.ml</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Focus on solving real business tasks: Your First Implementation of Machine Learning. #Coldstart.ml]]></title>
            <link>https://medium.com/shortcutnlp/focus-on-solving-real-business-tasks-your-first-implementation-of-machine-learning-coldstart-ml-52573d33624?source=rss-bc2291698c8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/52573d33624</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akira Takezawa]]></dc:creator>
            <pubDate>Mon, 18 Mar 2019 17:52:44 GMT</pubDate>
            <atom:updated>2019-03-18T17:52:44.748Z</atom:updated>
            <content:encoded><![CDATA[<h4>ビジネスにおける機械学習の活用事例</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hGpSEiT04ww2AyaliJfQwg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/wOJXPl-P4-w?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Jonathan Klok</a> on <a href="https://unsplash.com/search/photos/start?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Menu</h3><ol><li>Why can we use Regression for Prediction?</li><li>A process of ML implementation</li><li>Validation Metrics in Regression Tasks</li><li>実装: Boston House Price Prediction</li><li>Conclusion &amp; References</li></ol><p>— — — — —</p><h3>1. Why can we use Regression for Prediction?</h3><p>なぜ線形回帰か</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zX73aD6JCtgLGUXYohZARA.jpeg" /></figure><h3>Concept</h3><p>考え方</p><p><em>最終的に決めたい判断 y がある。ユーザーのデモグラフィック情報やサービス内での行動ログを判断材料 x として、yを予測しよう</em></p><p>回帰問題と分類問題どちらにも適応できる</p><p><strong>分類問題</strong></p><p><em>最終的に求めたい答えがカテゴリのとき</em></p><p>年齢,性別,年収…という特徴(x)をもつユーザーAが債務破綻に陥る</p><ul><li>特徴(x)：x1, x2, x3 ……</li><li>債務破綻率(y)：</li><li>前提：y &gt; 0.2 の時、クレジットカードを発行する</li></ul><p><strong>回帰問題</strong></p><p><em>最終的に求めたい答えが数値の時</em></p><h3>数学的な解釈</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*4qXM6uQt9Op5grmKkaUbQQ.gif" /><figcaption><a href="https://hackernoon.com/visualizing-linear-regression-with-pytorch-9261f49edb09">https://hackernoon.com/visualizing-linear-regression-with-pytorch-9261f49edb09</a></figcaption></figure><blockquote>y = mx + c</blockquote><ul><li>m = slope 傾き</li><li>c = intercept 切片</li></ul><p>切片：特徴がないときにも存在するからバイアス</p><p>デフォルトで値が設定されている</p><p>予測（ライン）と実際値の誤差が最小になる線を引く</p><p>(x, y)</p><p>x = [ 1,2,3,4,5 ] ( Ave(x) = 3 )</p><p>y = [ 2,4,6,4,8 ] ( Ave(y) = 4.8 )</p><p>傾きm：xの移動に対してyがどう変化していくか（重み）</p><p>切片c：xに値がない時（x=0）のyの初期値（バイアス）</p><p>m=( x-A(x) )( y-A(y) )/( x-A(x) )**2</p><p>AとBの誤差を縮めたい 全体最適にするため誤差の輪が最小になる公式を作る</p><ul><li>A: フォーミュラにxnをぶち込んで出る戻り値y</li><li>B: xn実際の戻り値y</li></ul><p>距離を相対化する方法：２乗 or 絶対値</p><p>トレーニングデータから公式を作る（m,cを決める）</p><p>テストデータで分類または回帰の精度を検証する</p><p>精度検証に使われる指標</p><p>— — — — —</p><h3>2. A process of ML implementation</h3><p>機械学習ワークフロー</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-4sTsXMzGJDXMDnOZEdz3w.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/lCYoIM-fbuU?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Jo Szczepanska</a> on <a href="https://unsplash.com/search/photos/process?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><ol><li>yを決める</li><li>xを決める（次元削除, PCA）</li><li>データをトレーニング/テスト用に分割する</li><li>トレーニングする（How: 最小誤差）</li><li>テストする（How: 評価指標）</li><li>実装して検証する</li></ol><p>— — — — —</p><h3>3. Validation Metrics in Regression Tasks</h3><p>回帰タスクにおける評価指標</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mOfFxGkE0FP-eQ30xoYxlQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/dQf7RZhMOJU?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Fleur Treurniet</a> on <a href="https://unsplash.com/search/photos/measure?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><ol><li>R2 (R-squared, coefficient of determination) 決定係数</li><li>MAE (Mean Absolute Error) 平均絶対誤差</li><li>MSE (Mean Squared Error) 平均絶対誤差</li><li>RMSE (Root Mean Squared Error) 平均平方絶対誤差</li></ol><p>— — — — —</p><h3>4. 実装: Boston House Price Prediction</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s4Vla6WhNpe1y-8WlOvuhQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/561igiTyvSk?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Jesse Roberts</a> on <a href="https://unsplash.com/search/photos/house?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><strong>0. Data Preparation: “</strong><a href="https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html"><strong>Boston House Price dataset</strong></a><strong>” which is scikit learn built-in dataset</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*Cl7y0q0lraNt-KmrVzv2EA.png" /><figcaption>DataFrame of Boston House Price Dataset</figcaption></figure><p><strong>1. Define Target Variable as y and Features as X 目的/説明変数の定義</strong></p><ul><li><strong>Target Variable: </strong>House price</li><li><strong>Features: </strong>average number of rooms, crime rate of district, property tax rate…. (we have 14 features)</li></ul><p>This means we are going to predict { y: House Price } by depending on { X: Features }.</p><p><strong>2. Feature Selection 特徴量の取捨選択</strong></p><p>We need to think if we really need each feature for prediction. In machine learning, we wanna use only meaning full features for Target.</p><p>There are two ways to decide features:</p><ol><li>Choosing by human hands: troublesome and risky</li><li>Dimensionality Reduction by the statistical methods: PCA, SVD etc…</li></ol><p>About <a href="https://bigdata-madesimple.com/decoding-dimensionality-reduction-pca-and-svd/">2.Dimensionality Reduction</a>, it will be a huge topic. So this time I will choose features manually for feeding to model depending on the degree of correlation. You can visualize correlations by plotting distribution like below.</p><pre># Using seaborn as visualization library<br>sns.jointplot(x=&quot;Crime Rate&quot;,y=&quot;House Price&quot;,data=df,kind=&quot;reg&quot;)<br>sns.jointplot(x=&quot;House Age&quot;,y=&quot;House Price&quot;,data=df,kind=&quot;reg&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*Kf_jixTTgsbSHUAVZL_Jng.jpeg" /></figure><p><strong>3. Feature Normalization 特徴量の標準化</strong></p><p>We need this process for improving our prediction accuracy because of <a href="https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/normalize-data">statistical reason</a>.</p><p>As you can see above graphs, every feature has a different scale of their value. For instance, {crime rate: 0~0.100 }, {house age: 1~100 } and so on.</p><p>If it goes on like this, the largest value affects more on our prediction model. To get a better prediction, we can treat all features fairly by Normalization.</p><pre># Scikitlearn has functions for Normalization<br>from sklearn.preprocessing import MinMaxScaler<br>scaler = MinMaxScaler()</pre><pre># We will Normalize only features, not Target variable<br>scaler.fit(X)<br>scaler.transform(X)</pre><p><strong>List of Normalization 主要なスケール標準化手法 </strong>:</p><p>MinMaxScaler, StandardSclare、RobustSclaer、Normalize…</p><p><strong>4. Split data for training and testing machine learning model データ分割</strong></p><pre>from sklearn.model_selection import train_test_split</pre><pre>sklearn公式ドキュメント</pre><p><strong>5. Training and Testing your data 学習とテスト</strong></p><pre># Import Linear Regression Model from scikit learn<br>from sklearn.linear_model import LinearRegression</pre><pre># Building prediction model &quot;lr&quot; by training data X, y<br>lr = LinearRegression().fit(X_train, y_train)</pre><pre># y_predict is y which feed X_test to lr<br>y_predict = lr.predict(X_test)</pre><pre># Evaluate test accuracy by test data X, y<br>lr.score(X_test, y_test)</pre><p><strong>6. Evaluate model by Metrics for Regression Problem 評価</strong></p><p>To identify the accuracy of House Price prediction model, we measure each distance between prediction model by training data and actual value from test data.</p><p>You can measure the distance(=accuracy) in several ways, and we have handy functions for it in scikit-learn.</p><pre># MAE (Mean Absolute Error)<br>from sklearn.metrics import mean_absolute_error<br>mean_absolute_error(y_test, y_predict)<br>&gt;&gt;&gt; output: 3.2</pre><pre># MSE (Mean Squared Error)<br>from sklearn.metrics import mean_squared_error<br>mean_squared_error(y_test, y_predict)<br>&gt;&gt;&gt; output: 24.90</pre><pre># RMSE (Root Mean Squared Error)<br>import numpy as np<br>np.sqrt(mean_squared_error(y_test, y_predict))<br>&gt;&gt;&gt; output: 4.90</pre><pre># R2 (coefficient of determination)<br>from sklearn.metrics import r2_score<br>r2_score(y_test, y_predict)<br>&gt;&gt;&gt; output: 0.70</pre><p>— — — — —</p><h3>Conclusion &amp; References</h3><ul><li>あるタスクに対して何を特徴とするかがまず大切</li><li>特徴を定性的にジャッジする方法と定量的に最適化する方法がある</li><li>上記の最小化＝モデルの最適化という考え</li></ul><h4>References</h4><ul><li><a href="https://www.oreilly.com/library/view/practical-statistics-for/9781491952955/ch04.html">Practical Statistics for Data Scientists</a></li><li><a href="https://statisticsbyjim.com/regression/predictions-regression/">Making Predictions with Regression Analysis - Statistics By Jim</a></li><li><a href="https://towardsdatascience.com/machine-learning-project-predicting-boston-house-prices-with-regression-b4e47493633d">Machine Learning Project: Predicting Boston House Prices With Regression</a></li><li><a href="https://rstudio-pubs-static.s3.amazonaws.com/364346_811c9012a14847428c9b1fc1e956431a.html">Boston Housing - Price Prediction</a></li><li><a href="https://medium.com/@haydar_ai/learning-data-science-day-9-linear-regression-on-boston-housing-dataset-cd62a80775ef">Learning Data Science: Day 9 - Linear Regression on Boston Housing Dataset</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52573d33624" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shortcutnlp/focus-on-solving-real-business-tasks-your-first-implementation-of-machine-learning-coldstart-ml-52573d33624">Focus on solving real business tasks: Your First Implementation of Machine Learning. #Coldstart.ml</a> was originally published in <a href="https://medium.com/shortcutnlp">Coldstart.ml</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to implement Seq2Seq LSTM Model in Keras #ShortcutNLP]]></title>
            <link>https://medium.com/data-science/how-to-implement-seq2seq-lstm-model-in-keras-shortcutnlp-6f355f3e5639?source=rss-bc2291698c8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f355f3e5639</guid>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Akira Takezawa]]></dc:creator>
            <pubDate>Mon, 18 Mar 2019 17:00:52 GMT</pubDate>
            <atom:updated>2021-03-08T13:05:44.740Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to implement Seq2Seq LSTM Model in Keras</h3><h4>If you got stuck with the Dimension problem, this is for you</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*sVg3CU8qDpjmLGBspTnFUw.png" /><figcaption><a href="https://www.techleer.com/articles/440-keras-deep-learning-for-python/">Keras: Deep Learning for Python</a></figcaption></figure><h3>Why do you need to read this?</h3><p>At the first time when I tried to implement seq2seq for Chatbot Task, I got stuck a lot of times especially about the Dimension of Input Data and Input layer of Neural Network Architecture.</p><p>Now I understand that unless you have a deep understanding of linear algebra concepts such as matrix and tensor, or how Keras API works, you’ll get errors continuously(and that’s so miserable!).</p><p>So in this article, I will explain the complete step-by-step guide of seq2seq for Keras, by checking the inputs and outputs in each process carefully.</p><p>In addition, I will provide some portable original functions which you can use in any other NLP projects for preprocessing.</p><p>Let&#39;s get started!</p><h3>Menu</h3><ol><li>What is Seq2Seq Text Generation Model?</li><li>Task Definition and Seq2Seq Modeling</li><li>Dimensions of Each Layer from Seq2Seq</li><li>Preprocessing of Seq2Seq (in Chatbot Case)</li><li>Simplest preprocessing of code: which you can use today!</li></ol><h3>1. What is Seq2Seq Text Generation Model?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LYGO4IxqUYftFdAccg5fVQ.png" /><figcaption>Fig A — Encoder-Decoder training architecture for NMT — image copyright@<a href="https://medium.com/u/c3f8c66f5451">Ravindra Kompella</a></figcaption></figure><blockquote>Seq2Seq is a type of Encoder-Decoder model using RNN. It can be used as a model for machine interaction and machine translation.</blockquote><p>By learning a large number of sequence pairs, this model generates one from the other. More kindly explained, the I/O of Seq2Seq is below:</p><ul><li>Input: sentence of text data e.g. “How are you doing?”</li><li>Output: sentence of text data as well e.g. “Not so bad.”</li></ul><p>And here we have examples of business applications with Seq2Seq:</p><ul><li>A chatbot (you can find it from my <a href="https://github.com/samurainote/chatbot_slack_keras">GitHub</a>)</li><li>Machine Translation (you can find from my <a href="https://github.com/samurainote/seq2seq_translate_slackbot">GitHub</a>)</li><li>Question Answering</li><li>Abstract Text Summarization (you can find from my <a href="https://github.com/samurainote/Text_Summarization_using_Bidirectional_LSTM">GitHub</a>)</li><li>Text Generation (you can find from my <a href="https://github.com/samurainote/Text_Generation_using_GRU">GitHub</a>)</li></ul><p>If you want more information about Seq2Seq, here I have a recommendation from <a href="https://www.youtube.com/channel/UCXvHuBMbgJw67i5vrMBBobA">Machine Learning at Microsoft</a> on Youtube.</p><p>Now we could understand the versatility of this technique, so let’s take a look at the whole process!</p><p>— — — — —</p><h3>2. Task Definition and Seq2Seq Modeling</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*CkeGXClZ5Xs0MhBc7xFqSA.png" /><figcaption><a href="https://www.oreilly.com/library/view/deep-learning-essentials/9781785880360/b71e37fb-5fd9-4094-98c8-04130d5f0771.xhtml">https://www.oreilly.com/library/view/deep-learning-essentials/9781785880360/b71e37fb-5fd9-4094-98c8-04130d5f0771.xhtml</a></figcaption></figure><p>For training our seq2seq model, we will use <a href="https://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html">Cornell Movie — Dialogs Corpus Dataset</a> which contains over 220,579 conversational exchanges between 10,292 pairs of movie characters. And it involves 9,035 characters from 617 movies.</p><p>Here one of the conversations from the data set:</p><pre>Mike: <br>&quot;Drink up, Charley. We&#39;re ahead of you.&quot;</pre><pre>Charley: <br>&quot;I&#39;m not thirsty.&quot;</pre><p>Then we will input these pairs of conversations into Encoder and Decoder. So that means our Neural Network model has two input layers as you can see below.</p><p>This is our Seq2Seq Neural Network Architecture for this time:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5102c53613c09a4473834da56293e923/href">https://medium.com/media/5102c53613c09a4473834da56293e923/href</a></iframe><p>Let&#39;s visualize our Seq2Seq by using LSTM:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*7EUe3SzWJgDbJsI0BISIUQ.png" /><figcaption>copyright Akira Takezawa</figcaption></figure><h3>3. Dimensions of Each Layer from Seq2Seq</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*DRwOwEdpT9wygpfY1DVhVA.jpeg" /><figcaption><a href="https://bracketsmackdown.com/word-vector.html">https://bracketsmackdown.com/word-vector.html</a></figcaption></figure><p>The Black Box for “NLP newbie” is I think this:</p><blockquote>How each layer compiles data and change their Dimension of data?</blockquote><p>To make this clear, I will explain how it works in detail. The Layers can be broken down into 5 different parts:</p><ol><li>Input Layer (Encoder and Decoder)</li><li>Embedding Layer (Encoder and Decoder)</li><li>LSTM Layer (Encoder and Decoder)</li><li>Decoder Output Layer</li></ol><p>Let’s get started!</p><h3>1. Input Layer of Encoder and Decoder (2D-&gt;2D)</h3><ul><li><strong>Input Layer Dimension: 2D (sequence_length, None)</strong></li></ul><pre># 2D<br>encoder_input_layer = Input(shape=(sequence_length, ))<br>decoder_input_layer = Input(shape=(sequence_length, ))</pre><p><em>NOTE: sequence_length is MAX_LEN unified by padding in preprocessing</em></p><ul><li><strong>Input Data: 2D (sample_num, max_sequence_length)</strong></li></ul><pre># Input_Data.shape = (150000, 15)</pre><pre>array([[ 1, 32, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],<br>       [ 123, 56, 3, 34, 43, 345, 0, 0, 0, 0, 0, 0, 0],<br>       [ 3, 22, 1, 6543, 58, 6, 435, 0, 0, 0, 0, 0, 0],<br>       [ 198, 27, 2, 94, 67, 98, 0, 0, 0, 0, 0, 0, 0],<br>       [ 45, 78, 2, 23, 43, 6, 45, 0, 0, 0, 0, 0, 0]<br>        ], dtype=int32)</pre><p><em>NOTE: sample_num can be a length of training_data (150000)</em></p><ul><li><strong>Output Data: 2D</strong></li></ul><p><em>NOTE: </em>Input()<em> is used only for Keras tensor instantiations</em></p><p>— — — — —</p><h3>2. Embedding layer of Encoder and Decoder (2D-&gt;3D)</h3><ul><li><strong>Embedding Layer Dimension: 2D (sequence_length, vocab_size)</strong></li></ul><pre>embedding_layer = Embedding(input_dim = <strong>vocab_size</strong>,<br>                            output_dim = <strong>embedding_dimension</strong>, <br>                            input_length = <strong>sequence_length</strong>)</pre><p><em>NOTE: </em><strong><em>vocab_size</em></strong><em> is the number of unique words</em></p><ul><li><strong>Input Data: 2D (sequence_length, vocab_size)</strong></li></ul><pre># Input_Data.shape = (15, 10000)</pre><pre>array([[ 1, 1, 0, 0, 1, 0, ...... 0, 0, 1, 0, 0, 0, 0],<br>       [ 0, 0, 1, 0, 0, 1, ...... 0, 0, 0, 0, 0, 0, 1],<br>       [ 0, 1, 0, 0, 0, 0, ...... 0, 0, 1, 0, 0, 0, 0],<br>       [ 0, 1, 0, 0, 0, 1, ...... 0, 0, 0, 1, 0, 1, 0],<br>       [ 0, 0, 1, 0, 1, 0, ...... 0, 0, 1, 0, 1, 0, 0]<br>        ], dtype=int32)</pre><p><em>NOTE: Data should be a group of One-Hot Vector</em></p><ul><li><strong>Output Data: 3D (num_samples, sequence_length, embedding_dims)</strong></li></ul><pre># Output_Data.shape = (150000, 15, 50)</pre><pre>array([[[ 1, 1, 0, 0, ...... 0, 1, 0, 0],<br>        [ 0, 0, 1, 0, ...... 0, 0, 0, 1],<br>         ..., <br>         ..., <br>        [ 0, 1, 0, 0, ...... 1, 0, 1, 0],<br>        [ 0, 0, 1, 0, ...... 0, 1, 0, 0]], <br></pre><pre>       [[ 1, 1, 0, 0, ...... 0, 1, 0, 0],<br>        [ 0, 0, 1, 0, ...... 0, 0, 0, 1],<br>         ..., <br>         ..., <br>        [ 0, 1, 0, 0, ...... 1, 0, 1, 0],<br>        [ 0, 0, 1, 0, ...... 0, 1, 0, 0]],</pre><pre>        ....,] * 150000      , dtype=int32)</pre><p><em>NOTE: Data is word embedded in 50 dimensions</em></p><p>— — — — —</p><h3>3. LSTM layer of Encoder and Decoder (3D-&gt;3D)</h3><p>The tricky argument of the LSTM layer is these two:</p><blockquote>1. return_state:</blockquote><blockquote>Whether to return the last state along with the output</blockquote><blockquote><strong>2. return_sequences</strong>:</blockquote><blockquote>Whether the last output of the output sequence or a complete sequence is returned</blockquote><p>You can find a good explanation from <a href="https://machinelearningmastery.com/return-sequences-and-return-states-for-lstms-in-keras/"><strong>Understand the Difference Between Return Sequences and Return States for LSTMs in Keras</strong></a><strong> </strong>by <a href="https://machinelearningmastery.com/author/jasonb/"><strong>Jason Brownlee</strong></a>.</p><ul><li><strong>Layer Dimension: 3D (hidden_units, sequence_length, embedding_dims)</strong></li></ul><pre># HIDDEN_DIM = 20</pre><pre>encoder_LSTM = LSTM(HIDDEN_DIM, return_state=True)    encoder_outputs, state_h, state_c = encoder_LSTM(encoder_embedding)</pre><pre>decoder_LSTM = LSTM(HIDDEN_DIM, return_state=True, return_sequences=True)   <br>decoder_outputs, _, _ = decoder_LSTM(decoder_embedding, initial_state=[state_h, state_c])</pre><ul><li><strong>Input Data: 3D (num_samples, sequence_length, embedding_dims)</strong></li></ul><pre># Input_Data.shape = (150000, 15, 50)</pre><pre>array([[[ 1, 1, 0, 0, ...... 0, 1, 0, 0],<br>        [ 0, 0, 1, 0, ...... 0, 0, 0, 1],<br>         ..., <br>         ..., <br>        [ 0, 1, 0, 0, ...... 1, 0, 1, 0],<br>        [ 0, 0, 1, 0, ...... 0, 1, 0, 0]],</pre><pre>       [[ 1, 1, 0, 0, ...... 0, 1, 0, 0],<br>        [ 0, 0, 1, 0, ...... 0, 0, 0, 1],<br>         ..., <br>         ..., <br>        [ 0, 1, 0, 0, ...... 1, 0, 1, 0],<br>        [ 0, 0, 1, 0, ...... 0, 1, 0, 0]],</pre><pre>          ....,] * 150000      , dtype=int32)</pre><p><em>NOTE: Data is word embedded in 50 dimensions</em></p><ul><li><strong>Output Data: 3D (num_samples, sequence_length, hidden_units)</strong></li></ul><pre># HIDDEN_DIM = 20<br># Output_Data.shape = (150000, 15, 20)</pre><pre>array([[[ 0.0032, 0.0041, 0.0021, .... 0.0020, 0.0231, 0.0010],<br>        [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>         ..., <br>         ..., <br>        [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>        [ 0.0021, 0.0065, 0.0008, .... 0.0089, 0.0043, 0.0024]],</pre><pre>        [ 0.0032, 0.0041, 0.0021, .... 0.0020, 0.0231, 0.0010],<br>        [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>         ..., <br>         ..., <br>        [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>        [ 0.0021, 0.0065, 0.0008, .... 0.0089, 0.0043, 0.0024]],</pre><pre>         ....,] * 150000      ,   dtype=int32)</pre><p><em>NOTE: Data reshaped by LSTM as a hidden layer in 20 dimensions</em></p><p><strong>Additional Information:</strong></p><p>If return_state = False and return_sequences = False :</p><ul><li><strong>Output Data: 2D (num_sample, hidden_units)</strong></li></ul><pre># HIDDEN_DIM = 20<br># Output_Data.shape = (150000, 20)</pre><pre>array([[ 0.0032, 0.0041, 0.0021, .... 0.0020, 0.0231, 0.0010],<br>       [ 0.0076, 0.0767, 0.0761, .... 0.0098, 0.0065, 0.0076],<br>         ..., <br>         ..., <br>       [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>       [ 0.0021, 0.0065, 0.0008, .... 0.0089, 0.0043, 0.0024]]<br>       , dtype=float32)</pre><p>— — — — —</p><h3>4. Decoder Output Layer (3D-&gt;2D)</h3><ul><li><strong>Output Layer Dimension: 2D (sequence_length, vocab_size)</strong></li></ul><pre>outputs = TimeDistributed(Dense(VOCAB_SIZE, activation=&#39;softmax&#39;))(decoder_outputs)</pre><p><em>NOTE: TimeDistributedDense</em>layer <em>allows us to apply a layer to every temporal slice of an input</em></p><ul><li><strong>Input Data: 3D (num_samples, sequence_length, hidden_units)</strong></li></ul><pre># HIDDEN_DIM = 20<br># Input_Data.shape = (150000, 15, 20)</pre><pre>array([[[ 0.0032, 0.0041, 0.0021, .... 0.0020, 0.0231, 0.0010],<br>        [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>         ..., <br>         ..., <br>        [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>        [ 0.0021, 0.0065, 0.0008, .... 0.0089, 0.0043, 0.0024]],</pre><pre>[ 0.0032, 0.0041, 0.0021, .... 0.0020, 0.0231, 0.0010],<br>        [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>         ..., <br>         ..., <br>        [ 0.0099, 0.0007, 0.0098, .... 0.0038, 0.0035, 0.0026],<br>        [ 0.0021, 0.0065, 0.0008, .... 0.0089, 0.0043, 0.0024]],</pre><pre>....,] * 150000      ,   dtype=int32)</pre><p><em>NOTE: Data reshaped by LSTM as a hidden layer in 20 dimensions</em></p><ul><li><strong>Output Data: 2D (sequence_length, vocab_size)</strong></li></ul><pre># Output_Data.shape = (15, 10000)</pre><pre>array([[ 1, 1, 0, 0, 1, 0, ...... 0, 0, 1, 0, 0, 0, 0],<br>       [ 0, 0, 1, 0, 0, 1, ...... 0, 0, 0, 0, 0, 0, 1],<br>       [ 0, 1, 0, 0, 0, 0, ...... 0, 0, 1, 0, 0, 0, 0],<br>       [ 0, 1, 0, 0, 0, 1, ...... 0, 0, 0, 1, 0, 1, 0],<br>       [ 0, 0, 1, 0, 1, 0, ...... 0, 0, 1, 0, 1, 0, 0]<br>        ], dtype=int32)</pre><p>After Data passed this Fully Connected Layer, we use <strong>Reversed Vocabulary </strong>which I will explain later to convert from One-Hot Vector into Word Sequence.</p><p>— — — — —</p><h3>4. Entire Preprocess of Seq2Seq (in Chatbot Case)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*ZX16gz2plU0AgIPDMPOYKA.png" /><figcaption>Creating A Language Translation Model Using Sequence To Sequence Learning Approach</figcaption></figure><p>Before jumping on preprocessing of Seq2Seq, I wanna mention about this:</p><blockquote>We need some Variables to define the Shape of our Seq2Seq Neural Network on the way of Data preprocessing</blockquote><ol><li>MAX_LEN: to unify the length of the input sentences</li><li>VOCAB_SIZE: to decide the dimension of sentence’s one-hot vector</li><li>EMBEDDING_DIM: to decide the dimension of Word2Vec</li></ol><p>— — — — —</p><p><strong>Preprocessing for Seq2Seq</strong></p><p>OK, please put this information on your mind, let’s start to talk about preprocessing. The whole process could be broken down into 8steps:</p><ol><li>Text Cleaning</li><li>Put &lt;BOS&gt; tag and &lt;EOS&gt; tag for decoder input</li><li>Make Vocabulary (VOCAB_SIZE)</li><li>Tokenize Bag of words to Bag of IDs</li><li>Padding (MAX_LEN)</li><li>Word Embedding (EMBEDDING_DIM)</li><li>Reshape the Data depends on neural network shape</li><li>Split Data for training and validation, testing</li></ol><p>— — — — —</p><h3><strong>1. Text Cleaning</strong></h3><ul><li><strong>Function</strong></li></ul><p>I always use this my own function to clean text for Seq2Seq:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0db307620796bb42fe8911d4a61f8b2d/href">https://medium.com/media/0db307620796bb42fe8911d4a61f8b2d/href</a></iframe><ul><li><strong>Input</strong></li></ul><pre># encoder input text data</pre><pre>[&quot;Drink up, Charley. We&#39;re ahead of you.&quot;,<br> &#39;Did you change your hair?&#39;,<br> &#39;I believe I have found a faster way.&#39;]</pre><ul><li><strong>Output</strong></li></ul><pre># encoder input text data</pre><pre>[&#39;drink up charley we are ahead of you&#39;,<br> &#39;did you change your hair&#39;,<br> &#39;i believe i have found a faster way&#39;]</pre><p>— — — — —</p><h3>2. Put &lt;BOS&gt; tag and &lt;EOS&gt; tag for decoder input</h3><ul><li><strong>Function</strong></li></ul><p>&lt;BOS&gt; means “Begin of Sequence”, &lt;EOS&gt; means “End of Sequence”.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/58ff0d385e0835663e64a575e2309ad0/href">https://medium.com/media/58ff0d385e0835663e64a575e2309ad0/href</a></iframe><ul><li><strong>Input</strong></li></ul><pre># decoder input text data</pre><pre>[[&#39;with the teeth of your zipper&#39;,<br>  &#39;so they tell me&#39;,<br>  &#39;so  which dakota you from&#39;],,,,]</pre><ul><li><strong>Output</strong></li></ul><pre># decoder input text data</pre><pre>[[&#39;&lt;BOS&gt; with the teeth of your zipper &lt;EOS&gt;&#39;,<br>  &#39;&lt;BOS&gt; so they tell me &lt;EOS&gt;&#39;,<br>  &#39;&lt;BOS&gt; so  which dakota you from &lt;EOS&gt;&#39;],,,,]</pre><p>— — — — —</p><h3>3. Make Vocabulary (VOCAB_SIZE)</h3><ul><li><strong>Function</strong></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc6f39204d026346bb64d3226172028c/href">https://medium.com/media/bc6f39204d026346bb64d3226172028c/href</a></iframe><ul><li><strong>Input</strong></li></ul><pre># Cleaned texts</pre><pre>[[&#39;with the teeth of your zipper&#39;,<br>  &#39;so they tell me&#39;,<br>  &#39;so  which dakota you from&#39;],,,,]</pre><ul><li><strong>Output</strong></li></ul><pre>&gt;&gt;&gt; word2idx</pre><pre>{&#39;genetically&#39;: 14816,<br> &#39;ecentus&#39;: 64088,<br> &#39;houston&#39;: 4172,<br> &#39;cufflinks&#39;: 30399,<br> &quot;annabelle&#39;s&quot;: 23767,<br> .....} # 14999 words</pre><pre>&gt;&gt;&gt; idx2word</pre><pre>{1: &#39;bos&#39;,<br> 2: &#39;eos&#39;,<br> 3: &#39;you&#39;,<br> 4: &#39;i&#39;,<br> 5: &#39;the&#39;,<br> .....} # 14999 indexs</pre><p>— — — — —</p><h3>4. Tokenize Bag of words to Bag of IDs</h3><ul><li>Function</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/653074ec5ff5bd126a3dde15bdfe56f8/href">https://medium.com/media/653074ec5ff5bd126a3dde15bdfe56f8/href</a></iframe><ul><li><strong>Input</strong></li></ul><pre># Cleaned texts</pre><pre>[[&#39;with the teeth of your zipper&#39;,<br>  &#39;so they tell me&#39;,<br>  &#39;so  which dakota you from&#39;],,,,]</pre><ul><li><strong>Output</strong></li></ul><pre># decoder input text data</pre><pre>[[10, 27, 8, 4, 27, 1107, 802],<br> [3, 5, 186, 168],<br> [662, 4, 22, 346, 6, 130, 3, 5, 2407],,,,,]</pre><p>— — — — —</p><h3>5. Padding (MAX_LEN)</h3><ul><li><strong>Function</strong></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/079977f1df9073fef08c9662eaef13d8/href">https://medium.com/media/079977f1df9073fef08c9662eaef13d8/href</a></iframe><ul><li><strong>Input</strong></li></ul><pre># decoder input text data</pre><pre>[[10, 27, 8, 4, 27, 1107, 802],<br> [3, 5, 186, 168],<br> [662, 4, 22, 346, 6, 130, 3, 5, 2407],,,,,]</pre><ul><li><strong>Output</strong></li></ul><pre># MAX_LEN = 10<br># decoder input text data </pre><pre>array([[10, 27, 8, 4, 27, 1107, 802, 0, 0, 0],<br>       [3, 5, 186, 168, 0, 0, 0, 0, 0, 0],<br>       [662, 4, 22, 346, 6, 130, 3, 5, 2407, 0],,,,,]</pre><p>— — — — —</p><h3>6. Word Embedding (EMBEDDING_DIM)</h3><ul><li><strong>Function</strong></li></ul><p>We use Pretraind Word2Vec Model from Glove. We can create embedding layer with Glove with 3 steps:</p><ol><li>Call Glove file from XX</li><li>Create Embedding Matrix from our Vocabulary</li><li>Create Embedding Layer</li></ol><p>Let’s take a look!</p><ul><li><strong>Call Glove file from XX</strong></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/14458b4c5dbf2824875b01ffae96f304/href">https://medium.com/media/14458b4c5dbf2824875b01ffae96f304/href</a></iframe><ul><li><strong>Create Embedding Matrix from our Vocabulary</strong></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/784d219d425a1ca9fe933c7d980a69be/href">https://medium.com/media/784d219d425a1ca9fe933c7d980a69be/href</a></iframe><ul><li><strong>Create Embedding Layer</strong></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d5226699e42d308e61acd95bc8ddac4/href">https://medium.com/media/0d5226699e42d308e61acd95bc8ddac4/href</a></iframe><p>— — — — —</p><h3>7. Reshape the Data to a neural network shape</h3><ul><li><strong>Function</strong></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34b74f952a1be943cd1dc63b72d03771/href">https://medium.com/media/34b74f952a1be943cd1dc63b72d03771/href</a></iframe><ul><li><strong>Input</strong></li></ul><pre># MAX_LEN = 10<br># decoder input text data</pre><pre>array([[10, 27, 8, 4, 27, 1107, 802, 0, 0, 0],<br>       [3, 5, 186, 168, 0, 0, 0, 0, 0, 0],<br>       [662, 4, 22, 346, 6, 130, 3, 5, 2407, 0],,,,,]</pre><ul><li><strong>Output</strong></li></ul><pre># output.shape (num_samples, MAX_LEN, VOCAB_SIZE)<br># decoder_output_data.shape (15000, 10, 15000)</pre><pre>array([[[0., 0., 0., ..., 0., 0., 0.],<br>        [0., 0., 0., ..., 0., 0., 0.],<br>        [0., 0., 1., ..., 0., 0., 0.],<br>        ...,<br>        [1., 0., 0., ..., 0., 0., 0.],<br>        [1., 0., 0., ..., 0., 0., 0.],<br>        [1., 0., 0., ..., 0., 0., 0.]],</pre><pre>        ..., ], , dtype=float32)</pre><p>— — — — —</p><h3>8. Split Data for training and validation, testing</h3><ul><li><strong>Function</strong></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a8646e80d9bc6ab0d8da9939976d01e/href">https://medium.com/media/5a8646e80d9bc6ab0d8da9939976d01e/href</a></iframe><ul><li><strong>Input</strong></li></ul><pre>array([[[0., 0., 0., ..., 0., 0., 0.],<br>        [0., 0., 0., ..., 0., 0., 0.],<br>        [0., 0., 1., ..., 0., 0., 0.],<br>        ...,<br>        [1., 0., 0., ..., 0., 0., 0.],<br>        [1., 0., 0., ..., 0., 0., 0.],<br>        [1., 0., 0., ..., 0., 0., 0.]],</pre><pre>..., ], , dtype=float32)</pre><ul><li><strong>Output</strong></li></ul><pre>array([[[0., 0., 0., ..., 0., 0., 0.],<br>        [0., 0., 0., ..., 0., 0., 0.],<br>        [0., 0., 1., ..., 0., 0., 0.]],</pre><pre>..., ], , dtype=float32)</pre><p>Now we finished the entire preprocessing process before input the learning data into our LSTM model.</p><p>After this, you just have to feed them and wait for finishing the learning.</p><p>Thanks for reading, see you in the next article.</p><p>— — — — —</p><h3>References</h3><ul><li><a href="https://chunml.github.io/ChunML.github.io/project/Sequence-To-Sequence/">Creating A Language Translation Model Using Sequence To Sequence Learning Approach</a></li><li><a href="http://sujitpal.blogspot.com/2016/10/deep-learning-models-for-question.html">Deep Learning Models for Question Answering with Keras</a></li><li><a href="https://nextjournal.com/gkoehler/machine-translation-seq2seq-cpu">Machine Translation using Sequence-to-Sequence Learning</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f355f3e5639" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-implement-seq2seq-lstm-model-in-keras-shortcutnlp-6f355f3e5639">How to implement Seq2Seq LSTM Model in Keras #ShortcutNLP</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[#08 Gradient Descent Algorithm Optimization Recipes #ShortcutML]]></title>
            <link>https://medium.com/shortcutnlp/08-gradient-descent-algorithm-optimization-recipes-shortcutml-b18b2530a886?source=rss-bc2291698c8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b18b2530a886</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akira Takezawa]]></dc:creator>
            <pubDate>Mon, 18 Mar 2019 13:11:44 GMT</pubDate>
            <atom:updated>2019-03-18T18:02:31.467Z</atom:updated>
            <content:encoded><![CDATA[<h4>勾配降下法の最適化アルゴリズム</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3tHAT6xy8TolHgDDEtuXrg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/AmlkxHrYsio?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Todd Quackenbush</a> on <a href="https://unsplash.com/search/photos/swim?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Hola! Welcome to #ShortcutML Series! ML Cheat Note for everyone!</p><h3>Menu</h3><ol><li>The goal of all ML Implementation</li><li>Bias vs. Variance in Error Function</li><li>Underfitting vs. Overfitting</li><li>Gradient Descent Methods List</li><li>Optimization Algorithms of GD List</li></ol><h3>1. The goal of all ML Implementation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*16uR2JPzCtu2qSCGPe7kbw.jpeg" /><figcaption><a href="https://www.youtube.com/watch?v=b4Vyma9wPHo">https://www.youtube.com/watch?v=b4Vyma9wPHo</a></figcaption></figure><p>The process of ML implementation can be called</p><blockquote>The process of minimizing the number or size of Error between prediction and unknown future.</blockquote><p>The purpose of ML is almost always prediction for unknown futures with high accuracy and high speed, high continuity. They reply class or value, sometimes words and text.</p><p>Therefore as a machine learning engineer, the main concern should be <strong>“How can we reduce errors and realize accurate prediction?”</strong></p><p>The answer to this fundamental question is this topic!: <strong>Gradient Descent</strong></p><h3>2. Bias vs. Variance in Error Function</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*kADA5Q4al9DRLoXck6_6Xw.jpeg" /></figure><p>Error = Bias + Variance + Noise*</p><ul><li>Bias: Gap between training data and model</li><li>Variance: Complexity of model</li><li>Noise:</li></ul><h4>A trade-off between Bias and Variance</h4><blockquote>To decrease Bias, you need to increase Variance of model.</blockquote><blockquote>To decrease Variance, you have to accept growth of Bias.</blockquote><ul><li>model complexity from linear to a random tree</li><li><strong>Sweet spot!!! As low as possible!!</strong></li><li>noise is always there means 100% accuracy is impossible</li></ul><h3>3. Underfitting vs. Overfitting</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*ePMqfevdkn4GBgvsZKrCcw.jpeg" /><figcaption><a href="https://towardsdatascience.com/understanding-the-bias-variance-tradeoff-165e6942b229">https://towardsdatascience.com/understanding-the-bias-variance-tradeoff-165e6942b229</a></figcaption></figure><p><strong>Why does happen?</strong></p><p>The reason is occasional but mainly these two reasons:</p><ul><li>When The size of training data is small</li><li>When the model is too complex for the task</li></ul><p>Thus the solution for Underfitting and Overfitting could be below:</p><ul><li>Increase your amount of Data</li><li>Regularization</li></ul><h3>4. A list of Gradient Descent Methods</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*20qjl1C-G7t2xLghzfxhmg.jpeg" /><figcaption><a href="https://hackernoon.com/gradient-descent-aynk-7cbe95a778da">Gradient Descent: All You Need to Know</a></figcaption></figure><h4>What is Gradient Descent?</h4><p>The gradient descent method is a method for minimizing the objective function parameterized by parameters of the model and is realized by updating the parameter in the direction opposite to the gradient direction of the objective function.</p><p>The learning rate determines how large steps are taken to reach the (local) minimum.</p><p>In other words, it is assumed that there is a surface made by the objective function, moving according to the downward slope of the surface until it reaches this valley.</p><p>Here is a well-known top 3 method:</p><ol><li>Stochastic Gradient Descent</li><li>Mini-batch Gradient Descent</li><li>Batch Gradient Descent</li></ol><h4>4–2. Comparison of gradient descent methods</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*3L2t1Da4M3ztbB0I1Torhw.png" /><figcaption>Linear Regression- Deep View(Part 1)</figcaption></figure><h4>1. Stochastic Gradient Descent</h4><p>The stochastic gradient descent method shuffles learning data, randomly extracts one from the learning data, calculates the error, and updates the parameters. Although it is not as accurate as the gradient descent method but re-learned only with the increased learning data (the initial value of the weight vector is diverted from the previous learning result), the calculation amount of re-learning is overwhelmingly low.</p><h3>5. A list of Optimization Algorithms of GD</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*XVFmo9NxLnwDr3SxzKy-rA.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*SjtKOauOXFVjWRR7iCtHiA.gif" /><figcaption><a href="http://ruder.io/optimizing-gradient-descent/">http://ruder.io/optimizing-gradient-descent/</a></figcaption></figure><p><strong>Keyword:</strong></p><ul><li>Global vs Minimum</li><li>Learning Rate</li></ul><h4>Which optimizer to choose?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*d5bICWZAuEAYfcdhQtpHEA.png" /><figcaption>SGD &gt; Adam?? Which One Is The Best Optimizer: Dogs-VS-Cats Toy Experiment</figcaption></figure><p>Here I listed several Optimization Algorithm for Gradient Descent:</p><ol><li>Momentum</li><li>Adagrad</li><li>Adadelta</li><li>RMSprop</li><li>Adam</li></ol><p>Let’s take a look all of them!</p><h4>1. Momentum</h4><p>The Momentum term increases towards the dimension where the gradient points in the same direction. And the gradient decreases the update towards the changing direction. As a result, the convergence speeds up and vibration can be suppressed.</p><h4>2. Adagrad</h4><p>Adagrad adapts the learning rate to the parameters. And we will perform larger updates for rare parameters and smaller updates for frequent parameters. For this reason, it is suitable for handling sparse data.</p><h4>3. Adadelta</h4><p>Adadelta is an evolution of Adagrad that explores ways to prevent a rapid and monotonous decline in learning rates. Adadelta does not accumulate the square of the gradient at all past time steps but limits the area where the past gradient is accumulated to certain constants.</p><h4>4. RMSprop</h4><p>RMSprop is an algorithm that’s used for full-batch optimization. It tries to resolve the problem that gradients may vary widely in magnitudes. Some gradients may be tiny and others may be huge, which result in a very difficult problem, trying to find a single global learning rate for the algorithm.</p><h4>5. Adam</h4><p>Adam (Adaptive Moment Estimation) is another way to calculate and adapt the learning rate for each parameter. Adadelta and RMSprop accumulated the exponential decay average of the squares of past gradients.</p><p>In addition to this, Adam holds the exponential decay mean of the past slope in the same way. It is a method similar to Momentum.</p><p>— — —</p><h3>References</h3><ul><li><a href="https://towardsdatascience.com/10-gradient-descent-optimisation-algorithms-86989510b5e9">10 Gradient Descent Optimisation Algorithms</a></li><li><a href="https://towardsdatascience.com/understanding-the-bias-variance-tradeoff-165e6942b229">Understanding the Bias-Variance Tradeoff</a></li><li><a href="https://towardsdatascience.com/gradient-descent-algorithm-and-its-variants-10f652806a3">Gradient Descent Algorithm and Its Variants</a></li><li><a href="https://www.hellocybernetics.tech/entry/2017/01/24/100415">バイアス-バリアンス分解：機械学習の性能評価 - HELLO CYBERNETICS</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b18b2530a886" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shortcutnlp/08-gradient-descent-algorithm-optimization-recipes-shortcutml-b18b2530a886">#08 Gradient Descent Algorithm Optimization Recipes #ShortcutML</a> was originally published in <a href="https://medium.com/shortcutnlp">Coldstart.ml</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Don’t waste your time to find the best article about Machine Learning #ShortcutML]]></title>
            <link>https://medium.com/@akiratakezawa/dont-waste-your-time-to-find-the-best-article-about-machine-learning-shortcutml-b4def1b88668?source=rss-bc2291698c8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4def1b88668</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akira Takezawa]]></dc:creator>
            <pubDate>Sat, 16 Feb 2019 13:15:32 GMT</pubDate>
            <atom:updated>2019-02-16T17:31:44.294Z</atom:updated>
            <content:encoded><![CDATA[<h4>機械学習周辺のベストリソース</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EUxEE0GE0DqKGIObtzAXQA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/m1deYbnXPmM?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Joshua Rawson-Harris</a> on <a href="https://unsplash.com/search/photos/article?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>King</h3><p>Machine Learning Mastery</p><p><a href="https://machinelearningmastery.com/blog/">Gradient Descent With AdaGrad From Scratch - Machine Learning Mastery</a></p><p>Sebastian Ruder</p><p><a href="http://ruder.io/deep-learning-nlp-best-practices/">http://ruder.io/deep-learning-nlp-best-practices/</a></p><h3>On Medium</h3><h3>Tech Blog</h3><p>Google</p><p><a href="https://blog.google/technology/ai/">AI</a></p><p>Amazon</p><p><a href="https://aws.amazon.com/jp/blogs/machine-learning/">AWS Machine Learning Blog</a></p><p>Uber</p><p><a href="https://eng.uber.com/tag/machine-learning/">Machine Learning Archives | Uber Engineering Blog</a></p><p>Airbnb</p><p><a href="https://medium.com/airbnb-engineering">The Airbnb Tech Blog</a></p><p>Netflix</p><p><a href="https://medium.com/netflix-techblog">Netflix TechBlog</a></p><h3>Japanese Tech Blog</h3><ul><li><a href="https://hironsan.hatenablog.com/archive/category/%E8%87%AA%E7%84%B6%E8%A8%80%E8%AA%9E%E5%87%A6%E7%90%86">自然言語処理 カテゴリーの記事一覧 - Ahogrammer</a></li><li><a href="https://gengo.ai/ja/articles/">記事 | Gengo AI</a></li><li><a href="https://techblog.zozo.com/archive/category/%E6%A9%9F%E6%A2%B0%E5%AD%A6%E7%BF%92">機械学習 カテゴリーの記事一覧 - ZOZO Technologies TECH BLOG</a></li><li><a href="https://tech-blog.abeja.asia/">ABEJA Arts Blog</a></li><li><a href="https://play.kikagaku.co.jp/">キカガク技術ブログへようこそ！ - キカガク技術ブログ ドキュメント</a></li></ul><h3>Educational Institution</h3><ul><li><a href="http://news.mit.edu/topic/machine-learning">Machine learning | MIT News</a></li><li><a href="http://aiweekly.co/">AI Weekly - AI News &amp; Leading Newsletter on Deep Learning &amp; Artificial Intelligence</a></li><li><a href="https://brighterion.com/">Home</a></li><li><a href="https://machinelearning-blog.com/">machinelearning-blog.com</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4def1b88668" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Comprehensive Overall picture of NLP in 2019 #ShortcutNLP]]></title>
            <link>https://medium.com/shortcutnlp/a-comprehensive-overall-picture-of-nlp-in-2019-shortcutnlp-e661b5726511?source=rss-bc2291698c8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e661b5726511</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akira Takezawa]]></dc:creator>
            <pubDate>Sat, 16 Feb 2019 08:17:13 GMT</pubDate>
            <atom:updated>2019-03-18T11:53:55.304Z</atom:updated>
            <content:encoded><![CDATA[<h4>自然言語処理における主要トピック</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DoZJFsImboN23jaFBCGKLQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/DhwKbmdlJa0?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">History in HD</a> on <a href="https://unsplash.com/search/photos/mural-history?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Hola! Welcome to #ShortcutNLP Series! Cheat Note for everyone!</p><h3>目次</h3><ol><li><strong>はじめに: 自然言語処理界隈の落とし穴</strong></li><li><strong>自然言語処理（NLP）のビジネス応用</strong></li><li><strong>データとしての言語の特徴と言語の構造</strong></li><li><strong>ディープラーニングと自然言語処理</strong></li><li><strong>自然言語処理（NLP）における要素技術</strong></li></ol><h3>1. <strong>はじめに: 自然言語処理界隈の落とし穴</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WNhtEVDWL55NJXbO3AFECA.png" /><figcaption>Natural Language Processing Saves Businesses Millions of Dollars</figcaption></figure><p><strong>1–1. この記事の目的</strong></p><p>自然言語処理にはタスクに関する用語（機械翻訳, 文章分類など）やその実現を支える要素技術（Word2Vec, LDAなど）, さらには言語自体の構造を表す名称（形態素解析, PoSなど）が無数に存在しており, 全体像を整理するのが大変という問題に直面した.</p><p>そこで今回はこの問題を解決して, いち早く自然言語処理タスクの実装に取りかかれるようにするためのソリューションとしてこの記事を書く.</p><p><strong>1–2. 自然言語処理の歴史</strong></p><p>自然言語処理における重要な発展は以下のように, ここ５年で全て起きている. もしあなたがGoogle翻訳の精度向上を感じているならばそれである.</p><ul><li>2013年：Word2Vecにより単語の意味付きベクトル化に成功</li><li>2014年：LSTMを用いたディープラーニングによるSeq2Seqの登場</li><li>2015年：Attentionの登場により、Seq2Seqの性能が劇的に向上</li></ul><p>実際, この3つの中身さえ押さえておけば, ビジネスにおける自然言語処理タスクの勘所は押さえたといっても過言ではない.</p><p><strong>1–3. 自然言語処理の典型手順</strong></p><p>典型的な手順は以下になる.</p><ol><li>文章の単語分解&amp;前処理 (stopwords, stemming, lemmatization …)</li><li>単語のベクトル化と意味の付与 (Word Embedding: word2vec, LDA …)</li><li>分類器/生成器のモデリング (seq2seq: Attention, Encoder-Decoder)</li><li>評価 (BELU, 交差エントロピー)</li></ol><p>コンピュータサイエンスに詳しくない人向けに大切な前提を書いておく. まずコンピュータは最終的に数値しか処理できない. 言い換えると機械学習ではテキストデータも画像データも数値化できて初めて応用できる技術なのである.</p><p>よって文章や文章を単語に分解して, 意味を持たせたまま数値化（ベクトル化）する技術であるWord2vecやDoc2vecが完成した段階で, NLP界隈における未来がだいぶ明るくなったのである.</p><h3>2. <strong>自然言語処理（NLP）のビジネス応用</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*2L3CfQQR9CVgOen9OYAyEg.png" /><figcaption>Is Line’s smart assistant Clova a real Alexa competitor?</figcaption></figure><p><strong>2–1. 日常に溢れる自然言語処理</strong></p><p>機械学習を応用した自然言語処理は, 僕たちがすでに当たり前のように使っているアプリケーションに応用されている.</p><p>今すぐに思い当たるのはこれら5つのサービス.</p><ul><li>Google自動翻訳</li><li>Gmailの自動返信メッセージの提案</li><li>各種サービスサイトにあるChat Bot</li><li>LINE Clovaなどのスマートスピーカー</li><li>SanSanのEightによる画像内の文字情報スキャン（画像認識か）</li></ul><p>— — —</p><p><strong>2–1. 自然言語処理タスクの3つの種類</strong></p><p>上記をより自然言語処理界隈で使われるTerminologyで表現すると, こういう形で高・低レイヤーのブレークダウンできるのではないだろうか.</p><p><strong>◯高レイヤー（Input/Outputデータによる仕分け）</strong></p><ol><li>テキスト to クラス分類（Classification）</li><li>テキスト to テキスト（Seq2Seq）</li><li>画像・音声 to テキスト（Multi-Modal）</li></ol><p><strong>◯低レイヤー（タスクによる仕分け）</strong></p><ol><li>機械翻訳 Machine Translation</li><li>対話システム Chat Bot</li><li>質問応答 Question Answering</li><li>文章要約 Text Summarization</li><li>音声認識 Speech Recognition</li><li>感情分析 Sentiment Analysis</li><li>スパム・フェイクニュース検知 Spam Filter &amp; Fake News Detection</li><li>ニュースコンテンツの分類 Document Classification</li><li>情報抽出 Information Extraction</li></ol><p><strong>◯ビジネス応用はあまりされていないが, 面白いタスク</strong></p><ul><li>文章生成</li><li>キャプション生成（画像→文章）</li></ul><p>どれもKerasを使えば実装できるものばかりで実際に動かしてみると本当におもしろい！</p><p>— — —</p><p><strong>2–3. ビジネスにおけるNLP応用の課題</strong></p><p>自然言語処理自体がビジネスにおいてキャッシュエンジンになる事例がまだ少ない. （一部SaaSでのチャットボット提供など例外あり）</p><p>事実, Googleの機械翻訳はイノベーションの塊なのだが, 無料で開放されており, 実は誰一人お金を払ってない.</p><p>そこで救世主として, テキストマイニングによる知識獲得が注目を浴びている. 例えばこのMediumにはかなりの洗練された知識が日々アップロードされており, この知識を活用することができれば明らかに普通の個人より賢い.</p><h3>3. 言語の構造と<strong>データとしての言語の特徴</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*1NPzLPf7pzxEV3F1XYd73w.png" /><figcaption><a href="https://ai-lab.scouty.co.jp/deep-learning/dependency-parsing/">https://ai-lab.scouty.co.jp/deep-learning/dependency-parsing/</a></figcaption></figure><p><strong>3–1. 言語が持つ構造</strong></p><p>人間が普段意識せずに話している言葉には、名詞や動詞といった<strong>「品詞」や</strong>主語・述語といった<strong>「文法」</strong>といった構造が存在する.</p><p>言い換えると人間は言葉を組み合わせて、文章を作り、文章となって初めて意味が生まれ、コミュニケーションが成立する.</p><p>機械（プログラム）が人間がインプットする自然言語を処理し、人間によるコミュニケーションを処理しようとする際には、こういった細かい構造を理解しておく必要がある.</p><p>実際の学習では、僕たちがスクラッチで書いたプログラムが「りんご=名詞」など知っているわけではない.</p><p>そのため、<strong>コーパス（Corpus）</strong>や<strong>シソーラス（Thesaurus）</strong>を使い、｛りんご：名詞｝といったようにという単語ごとにすでに品詞づけされている辞書を使ったマッチングが行われる.</p><p>機械学習のアルゴリズムを組み込んだプログラムはこの処理を順序に沿って行なっていく. 実際の手順は以下となる.</p><blockquote><strong>1. 形態素 Morphology</strong></blockquote><blockquote><strong>2. 構文 Syntacs</strong></blockquote><blockquote><strong>3. 意味 Semantics</strong></blockquote><p>それではそれぞれの手順の詳細と重要語句をまとめていく.</p><p>— — —</p><p><strong>3–2. トピック</strong></p><p><strong>1. 形態素解析 Morphology</strong></p><p>名詞や動詞, 形容詞など言葉のカテゴリや役割.</p><ul><li><strong>Part of Speech Tagging PoSタグ</strong></li></ul><p><strong>2. 構文解析 Syntactic Analysis</strong></p><p>主語や述語, 修飾語など文中での単語間における関係性.</p><ul><li><strong>Dependency Parsing 係り受け解析</strong></li></ul><p><strong>3. 意味解析 Semantic Analysis</strong></p><p>動物：猫, 犬など単語の意味的集まり.</p><ul><li>Topic Model（LDA, LSA, PLSA） 潜在意味解析</li><li>Word Enbedding（Word2vec, Doc2vec, LDA2vec）単語の埋め込み表現</li><li>Named Entity Recognition（NER） 固有表現抽出</li></ul><p><strong>4. 単語のトークン化 Simple Tokenization</strong></p><ul><li>Bag of Words 単語トークンの集合</li><li>N-gram 文字数単位での区切り</li><li>Tf-idf 単語の出現頻度とその希少性から重要語句を判別</li></ul><p><strong>5. 単語単位での前処理</strong></p><ol><li>Stemming 単語の語幹 dog, doggy -&gt; dog</li><li>Lemmatization 単語のカテゴリ is, are -&gt; be</li><li>Stop Words 頻出勝つ意味を与えない単語 a, on, for</li></ol><p>— — —</p><p><strong>3–3. 非構造化データ:テキストデータの特徴</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*U0120z3kXnCpBaaw9g2_kA.jpeg" /><figcaption>NLP Fundamentals: Where Humans Team Up With Machines To Help It Speak</figcaption></figure><p>一般的にテキストデータはその特徴として, 上記の図を見てわかるように, 以下のように定義されている.</p><ul><li>テキストデータ：離散型（階層的、再帰的）</li><li>画像、音声：連続型（DLが得意なのはこっち）</li></ul><p>よって, よりコンピュータが処理しやすい数値化されたデータに直すまでの工程が複雑になっている. これは画像認識の実装をしてもらえば感じることができる.</p><h3><strong>4. ディープラーニングと自然言語処理</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*qRNoLmSSwpp3Q2425vdpkA.jpeg" /><figcaption>Overview of Artificial Intelligence and Natural Language Processing</figcaption></figure><p><strong>4–1. 深層学習は自然言語処理に何をもたらしたか？</strong></p><p>シンプルに説明すると、以下の３つが挙げられるのではないか。</p><ol><li>文章の特徴表現が豊かになった（表現できる意味・文脈が広がった）</li><li>画像や音声など異なるデータを用いたMulti-Mordalな処理が可能に</li><li>文章内における単語間の階層関係を学習から導けるようになった</li></ol><p>1.はseq2seqによる機械翻訳やチャットボット 、2.はAmazon Alexaなどに見られる音声認識から画像からその画像の内容を描写するキャプション生成というタスクに見られる。</p><p>— — —</p><p><strong>4–2. 転移学習</strong></p><p>転送学習とは、大規模データセット上でモデルを訓練し、その事前訓練されたモデルを使用することで別の目の前にあるタスクの精度を高めるためにニューラルネットワークを学習をさせていくプロセスである。</p><p>もともとはコンピュータビジョンの分野において、ImageNetデータセットの登場により転送学習が徐々に普及した。</p><p>それでは、NLPにおいて転送学習をどのように利用していくか。</p><p>答えは、ニューラルネットワークが大規模なコーパスから単語同士の依存関係や文章同士の意味関係などのさまざまな言語現象を取得して学習できるようになる。</p><p>そして、言語に関してある種の”知識”を持ったニューラルネットワークは、感情分類などの自然言語処理タスクの処理精度を向上する。</p><p>具体的には、単語の埋め込み表現においてWord2Vec, GloVe, fastTextなどから事前学習向けの学習済みモデルの重みを使える。</p><h3>5. 各タスクと要素技術の関係</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Y6ceck6-5LWkTt_6ZCYmCw.png" /><figcaption>Named Entity Recognition (NER) with keras and tensorflow</figcaption></figure><p>ここでは, 自然言語処理タスクを個人で実装したい人向けに, 各ビジネスタスクに対してどのようなNLP理論が使われているのかをまとめる.</p><p><strong>Task1: 情報抽出</strong></p><ul><li>Element technology: 固有表現認識</li><li>Method: Deep Learning</li></ul><p><strong>Task2:レコメンドシステム</strong></p><ul><li>Element technology: <a href="https://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec">Item2vec</a></li><li><a href="https://www.kernix.com/blog/recommender-system-based-on-natural-language-processing_p10">Method:</a> Cosine Similarity</li></ul><p><strong>Task3: コンテンツ分類</strong></p><ul><li>Element technology: トピックモデル</li><li>Method: ML/DL or LDA, LSA</li></ul><p><strong>Task4: 文書要約（抽象型）</strong></p><ul><li>Element technology: Seq2Seq</li><li>Method: Attention LSTM</li></ul><p><strong>Task5: 機械翻訳</strong></p><ul><li>Element technology: Seq2seq</li><li>Method: <a href="https://machinelearningmastery.com/develop-neural-machine-translation-system-keras/">RNN</a>, encoder-decoder</li></ul><p><strong>Task6: 質問応答</strong></p><ul><li>Element technology: Beam Search</li><li>Method: LSTM</li></ul><p><strong>Task9: 音声認識（音声→文章）</strong></p><ul><li>Element technology: Beam Search</li><li>Method: CNN</li></ul><p><strong>Task10: キャプション生成（画像→文章）</strong></p><ul><li><a href="https://machinelearningmastery.com/develop-a-deep-learning-caption-generation-model-in-python/">Element technology:</a> Transfer Learning (VGG16 and Glove)</li><li><a href="https://machinelearningmastery.com/prepare-photo-caption-dataset-training-deep-learning-model/">Method:</a> RNN and LSTM</li></ul><h3>References</h3><ul><li><a href="https://www.upwork.com/hiring/for-clients/artificial-intelligence-and-natural-language-processing-in-big-data/">Overview of Artificial Intelligence and Natural Language Processing</a></li><li><a href="https://medium.com/intro-to-artificial-intelligence/entity-extraction-using-deep-learning-8014acac6bb8">Entity extraction using Deep Learning based on Guillaume Genthial work on NER</a></li><li><a href="https://towardsdatascience.com/named-entity-recognition-ner-meeting-industrys-requirement-by-applying-state-of-the-art-deep-698d2b3b4ede">Named Entity Recognition (NER), Meeting Industry’s Requirement by Applying state-of-the-art Deep…</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e661b5726511" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shortcutnlp/a-comprehensive-overall-picture-of-nlp-in-2019-shortcutnlp-e661b5726511">A Comprehensive Overall picture of NLP in 2019 #ShortcutNLP</a> was originally published in <a href="https://medium.com/shortcutnlp">Coldstart.ml</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[#ShortcutML: Gradient Descent Algorithm Optimization Recipe]]></title>
            <link>https://medium.com/@akiratakezawa/shortcutml-gradient-descent-algorithm-optimization-recipe-1e3edf815a5b?source=rss-bc2291698c8b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e3edf815a5b</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akira Takezawa]]></dc:creator>
            <pubDate>Fri, 15 Feb 2019 07:18:00 GMT</pubDate>
            <atom:updated>2019-03-18T12:08:23.034Z</atom:updated>
            <content:encoded><![CDATA[<h3>#08 Gradient Descent Algorithm Optimization Recipes #ShortcutML</h3><h4>勾配降下法の最適化アルゴリズム</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3tHAT6xy8TolHgDDEtuXrg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/AmlkxHrYsio?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Todd Quackenbush</a> on <a href="https://unsplash.com/search/photos/swim?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Hola! Welcome to #ShortcutML Series! ML Cheat Note for everyone!</p><h3>Menu</h3><ol><li>The goal of all ML Implementation</li><li>Bias vs. Variance in Error Function</li><li>Underfitting vs. Overfitting</li><li>Gradient Descent Methods List</li><li>Optimization Algorithms of GD List</li></ol><h3><strong>1. The goal of all ML Implementation</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*16uR2JPzCtu2qSCGPe7kbw.jpeg" /><figcaption><a href="https://www.youtube.com/watch?v=b4Vyma9wPHo">https://www.youtube.com/watch?v=b4Vyma9wPHo</a></figcaption></figure><p>The process of ML implementation can be called</p><blockquote>The process of minimizing the number or size of Error between prediction and unknown future.</blockquote><p>The purpose of ML is almost always prediction for unknown futures with high accuracy and high speed, high continuity. They reply class or value, sometimes words and text.</p><p>Therefore as a machine learning engineer, the main concern should be <strong>“How can we reduce errors and realize accurate prediction?”</strong></p><p>The answer to this fundamental question is this topic!: <strong>Gradient Descent</strong></p><h3>2. Bias vs. Variance in Error Function</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*kADA5Q4al9DRLoXck6_6Xw.jpeg" /></figure><p>Error = Bias + Variance + Noise*</p><ul><li>Bias: Gap between training data and model</li><li>Variance: Complexity of model</li><li>Noise:</li></ul><h4>A trade-off between Bias and Variance</h4><blockquote>To decrease Bias, you need to increase Variance of model.</blockquote><blockquote>To decrease Variance, you have to accept growth of Bias.</blockquote><ul><li>model complexity from linear to a random tree</li><li><strong>Sweet spot!!! As low as possible!!</strong></li><li>noise is always there means 100% accuracy is impossible</li></ul><h3>3. Underfitting vs. Overfitting</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*ePMqfevdkn4GBgvsZKrCcw.jpeg" /><figcaption><a href="https://towardsdatascience.com/understanding-the-bias-variance-tradeoff-165e6942b229">https://towardsdatascience.com/understanding-the-bias-variance-tradeoff-165e6942b229</a></figcaption></figure><p><strong>Why does happen?</strong></p><p>The reason is occasional but mainly these two reasons:</p><ul><li>When The size of training data is small</li><li>When the model is too complex for the task</li></ul><p>Thus the solution for Underfitting and Overfitting could be below:</p><ul><li>Increase your amount of Data</li><li>Regularization</li></ul><h3>4. A list of <strong>Gradient Descent Methods</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*20qjl1C-G7t2xLghzfxhmg.jpeg" /><figcaption><a href="https://hackernoon.com/gradient-descent-aynk-7cbe95a778da">Gradient Descent: All You Need to Know</a></figcaption></figure><h4>4–1. What is Gradient Descent?</h4><p>The gradient descent method is a method for minimizing the objective function parameterized by parameters of the model and is realized by updating the parameter in the direction opposite to the gradient direction of the objective function.</p><p>The learning rate determines how large steps are taken to reach the (local) minimum.</p><p>In other words, it is assumed that there is a surface made by the objective function, moving according to the downward slope of the surface until it reaches this valley.</p><p>Here is a well-known top 3 method:</p><ol><li>Stochastic Gradient Descent</li><li>Mini-batch Gradient Descent</li><li>Batch Gradient Descent</li></ol><h4>4–2. Comparison of gradient descent methods</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*3L2t1Da4M3ztbB0I1Torhw.png" /><figcaption>Linear Regression- Deep View(Part 1)</figcaption></figure><h4>1. Stochastic Gradient Descent</h4><p>The stochastic gradient descent method shuffles learning data, randomly extracts one from the learning data, calculates the error, and updates the parameters. Although it is not as accurate as the gradient descent method but re-learned only with the increased learning data (the initial value of the weight vector is diverted from the previous learning result), the calculation amount of re-learning is overwhelmingly low.</p><h3>5. A list of Optimization Algorithms of GD</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*XVFmo9NxLnwDr3SxzKy-rA.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*SjtKOauOXFVjWRR7iCtHiA.gif" /><figcaption><a href="http://ruder.io/optimizing-gradient-descent/">http://ruder.io/optimizing-gradient-descent/</a></figcaption></figure><p><strong>Keyword:</strong></p><ul><li>Global vs Minimum</li><li>Learning Rate</li></ul><h4>Which optimizer to choose?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*d5bICWZAuEAYfcdhQtpHEA.png" /><figcaption>SGD &gt; Adam?? Which One Is The Best Optimizer: Dogs-VS-Cats Toy Experiment</figcaption></figure><p>Here I listed several Optimization Algorithm for Gradient Descent:</p><ol><li>Momentum</li><li>Adagrad</li><li>Adadelta</li><li>RMSprop</li><li>Adam</li></ol><p>Let’s take a look all of them!</p><h4>1. Momentum</h4><p>The Momentum term increases towards the dimension where the gradient points in the same direction. And the gradient decreases the update towards the changing direction. As a result, the convergence speeds up and vibration can be suppressed.</p><h4>2. Adagrad</h4><p>Adagrad adapts the learning rate to the parameters. And we will perform larger updates for rare parameters and smaller updates for frequent parameters. For this reason, it is suitable for handling sparse data.</p><h4>3. Adadelta</h4><p>Adadelta is an evolution of Adagrad that explores ways to prevent a rapid and monotonous decline in learning rates. Adadelta does not accumulate the square of the gradient at all past time steps but limits the area where the past gradient is accumulated to certain constants.</p><h4>4. RMSprop</h4><p>RMSprop is an algorithm that’s used for full-batch optimization. It tries to resolve the problem that gradients may vary widely in magnitudes. Some gradients may be tiny and others may be huge, which result in a very difficult problem, trying to find a single global learning rate for the algorithm.</p><h4>5. Adam</h4><p>Adam (Adaptive Moment Estimation) is another way to calculate and adapt the learning rate for each parameter. Adadelta and RMSprop accumulated the exponential decay average of the squares of past gradients.</p><p>In addition to this, Adam holds the exponential decay mean of the past slope in the same way. It is a method similar to Momentum.</p><p>— — —</p><h3>References</h3><ul><li><a href="https://towardsdatascience.com/understanding-the-bias-variance-tradeoff-165e6942b229">Understanding the Bias-Variance Tradeoff</a></li><li><a href="https://towardsdatascience.com/gradient-descent-algorithm-and-its-variants-10f652806a3">Gradient Descent Algorithm and Its Variants</a></li><li><a href="https://www.hellocybernetics.tech/entry/2017/01/24/100415">バイアス-バリアンス分解：機械学習の性能評価 - HELLO CYBERNETICS</a></li><li><a href="http://ruder.io/optimizing-gradient-descent/">An overview of gradient descent optimization algorithms</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e3edf815a5b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>