<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Won Seob Seo on Medium]]></title>
        <description><![CDATA[Stories by Won Seob Seo on Medium]]></description>
        <link>https://medium.com/@wontheone1?source=rss-168108739f86------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*NCceEaAiYIHZhyMq</url>
            <title>Stories by Won Seob Seo on Medium</title>
            <link>https://medium.com/@wontheone1?source=rss-168108739f86------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 10:57:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@wontheone1/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Feature engineering for stock price movement]]></title>
            <link>https://medium.com/predict/feature-engineering-for-stock-price-movement-54669d4a7efc?source=rss-168108739f86------2</link>
            <guid isPermaLink="false">https://medium.com/p/54669d4a7efc</guid>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[kaggle-competition]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Won Seob Seo]]></dc:creator>
            <pubDate>Sat, 12 Jan 2019 16:09:59 GMT</pubDate>
            <atom:updated>2019-01-12T16:38:27.572Z</atom:updated>
            <content:encoded><![CDATA[<h3>Feature engineering for predicting stock price movement</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*5ZbY6qcN2go5XlvfFZSIHQ.jpeg" /></figure><p>Recently I’ve participated in <a href="https://www.kaggle.com/c/two-sigma-financial-news">a Kaggle competition</a> hosted by ‘Two sigma’. It is about using market and news data to predict stock price movement in 10 days. The <a href="https://www.kaggle.com/c/two-sigma-financial-news/data">detailed explanations of each columns</a> are well written by host and would be helpful to read in order to understand what kind of feature I am generating. Also, you can read whole source code and explanation from <a href="https://www.kaggle.com/wontheone1/various-experiments-on-feature-generation">my Kaggle kernel</a> as well.</p><p>First thing to note about this competition is that it is a kernel only competition, so there are various resource limitations such as 16GB RAM and no GPU available. So it is quite impossible to use all the data and add bunch of new features on top of that. So naturally you would limit the period of data to more recent ones rather than using too old data. Also, after feature importances analysis, columns that don’t help should go away.</p><p>First of all as with most time series data it helps to make window(rolling) statistics features. Even in real life, a lot of investors will check such things as what was the peak/lowest price in 1 year, is current price higher/lower than its 1 month moving average and all that (to enumerate a few ‘<a href="https://en.wikipedia.org/wiki/Moving_average">Moving Average</a>’, ‘<a href="https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average">Exponential Moving Average</a>’, ‘<a href="https://en.wikipedia.org/wiki/Bollinger_Bands">Bollinger Band</a>’, ‘<a href="https://www.fidelity.com/learning-center/trading-investing/technical-analysis/technical-indicator-guide/RSI">The Relative Strength Index</a>’, and ‘<a href="https://www.marketvolume.com/analysis/volume_ma.asp">Volume Moving Average</a>’). So it should make sense to create window statistics of 1 week, 2 week, 1 month and 1 year and so on. Use common and simple statistics like mean, median, max, min and exponentially weighted mean. Or be creative and come up with different things. Generate these window features based on numeric columns. In reality I generated lag features only on market data rather than news data (If I had more computational resources than I would have tried creating these from news data as well).</p><pre>BASE_FEATURES = [ &#39;returnsOpenPrevMktres10&#39;, &#39;returnsOpenPrevRaw10&#39;,<br>                  &#39;open&#39;, &#39;close&#39;]</pre><p>Also some of the numbers don’t mean much without the context of their usual range and their ratios to market mean (which also changes as time passes). What I mean is, if a stock’s opening price is 300 dollars today, is it oversold or overbought? If I was talking about Amazon this would be insanely cheap because nowadays (as of beginning of 2019) it trades around 1600$. But if I was talking about Tesla stock, it would be only moderately cheap since Tesla stock as of now traded around 310 ~ 350 $ range. Same principle applies to many other features such as raw returns (gain/loss which is not adjusted against any benchmark), (trading) volume. So it would make sense to generate the ratio between these features and market mean.</p><pre>def add_market_mean_col(market_df):<br>    daily_market_mean_df = market_df.groupby(&#39;time&#39;).mean()<br>    daily_market_mean_df = daily_market_mean_df[[&#39;volume&#39;, &#39;close&#39;]]<br>    merged_df = market_df.merge(daily_market_mean_df, left_on=&#39;time&#39;,<br>                                right_index=True, suffixes=(&quot;&quot;,&#39;_market_mean&#39;))<br>    merged_df[&#39;volume/volume_market_mean&#39;] = merged_df[&#39;volume&#39;] / merged_df[&#39;volume_market_mean&#39;]<br>    merged_df[&#39;close/close_market_mean&#39;] = merged_df[&#39;close&#39;] / merged_df[&#39;close_market_mean&#39;]<br>    return merged_df.reset_index(drop = True)<br><br>BASE_FEATURES = BASE_FEATURES + [&#39;volume&#39;, &#39;volume/volume_market_mean&#39;, &#39;close/close_market_mean&#39;]</pre><p>In a similar context, the ratio of opening price and closing price should tell us more than just raw closing/opening price.</p><pre>def generate_open_close_ratio(df):<br>    df[&#39;open/close&#39;] = df[&#39;open&#39;] / df[&#39;close&#39;]<br>    <br>BASE_FEATURES = BASE_FEATURES + [&#39;open/close&#39;]</pre><p>Likewise, we can generate the ratio of raw return values to its current opening/closing prices. Note that this is not a duplicate of residual return columns which are the returns after movement of the market as a whole has been accounted for, leaving only movements inherent to the instrument. The generated ratio is not adjusted by market movements it is just a ratio of its price delta versus its price.</p><pre>open_raw_cols = [&#39;returnsOpenPrevRaw1&#39;, &#39;returnsOpenPrevRaw10&#39;]<br>close_raw_cols = [&#39;returnsClosePrevRaw1&#39;, &#39;returnsClosePrevRaw10&#39;]<br><br>def raw_features_to_ratio_features(df):<br>    for col <strong>in</strong> open_raw_cols:<br>        df[col + &#39;/open&#39; ] = df[col] / df[&#39;open&#39;]<br>    for col <strong>in</strong> close_raw_cols:<br>        df[col + &#39;/close&#39;] = df[col] / df[&#39;close&#39;]<br><br>BASE_FEATURES = BASE_FEATURES + [&#39;returnsClosePrevRaw1/close&#39;, &#39;returnsClosePrevRaw10/close&#39;, &#39;returnsOpenPrevRaw1/open&#39;, &#39;returnsOpenPrevRaw10/open&#39;]</pre><p>The previously mentioned window statistics feature is generated based on the BASE_FEATURES we gathered. And merged on time and assetCode(which is like the id column of the market data)</p><pre>new_df = generate_features(market_train_df)<br>market_train_df = pd.merge(market_train_df, new_df, how = &#39;left&#39;, on = [&#39;time&#39;, &#39;assetCode&#39;])</pre><p>Because of the memory limitation I dropped many columns from news data. However, considering that the name of this competition is called “Using News to Predict Stock Movements”, I thought I should make use of the news data in some way. Also thinking about reality, a lot of traders and machines react to positive/negative news so it makes sense that news affects stock prices. So I decided that the most interesting columns for me are ‘sentimentClass’, ‘sentimentNegative’, ‘sentimentNeutral’ and ‘sentimentPositive’. ‘sentimentClass’ is counted (how many of news were neutral, positive, negative), the rest are taken as mean (mean of over all neutralness etc). Note that ‘assetName’ column acts as id column for news data. ‘assetCodes’ have list of asset codes and sometimes there are few dozens of codes per single news. This would makes sense if we examine some of the examples of ‘assetCodes’ and ‘assetName’.</p><pre>news_train_df.head(100)[[&#39;assetCodes&#39;, &#39;assetName&#39;]]</pre><p>From results I can see that one news relating to the asset name of ‘Microsoft Corp’ has related ‘assetCodes’ of ‘{‘MSFT.O’, ‘MSFT.F’, ‘MSFT.DE’, ‘MSFT.OQ’}’. These codes are just Microsoft stocks on different stock exchanges. So now it is clear that if one news is related to an asset name all related asset codes will be affected. Also merging table on ‘assetName’ is practically much easier because it is single name of an asset in both market and news data (On the contrary market data has ‘assetCode’ which has a single asset code and news data has ‘assetCodes’ which has more than one asset codes). So now let’s transform sentiment column accordingly and merge market and news data.</p><pre>def merge_with_news_data(market_df, news_df):<br>    news_df[&#39;firstCreated&#39;] = news_df.firstCreated.dt.hour<br>    news_df[&#39;assetCodesLen&#39;] = news_df[&#39;assetCodes&#39;].map(lambda x: len(eval(x)))<br>    news_df[&#39;asset_sentiment_count&#39;] = news_df.groupby([&#39;assetName&#39;, &#39;sentimentClass&#39;])[&#39;firstCreated&#39;].transform(&#39;count&#39;)<br>    kcol = [&#39;time&#39;, &#39;assetName&#39;]<br>    news_df = news_df.groupby(kcol, as_index=False).mean()<br>    market_df = pd.merge(market_df, news_df, how=&#39;left&#39;, on=kcol, suffixes=(&quot;&quot;, &quot;_news&quot;))<br>    return market_df<br><br>market_train_df = merge_with_news_data(market_train_df, news_train_df)</pre><p>Now that I have all the features I want to use let’s train some models. I used 3 models from 3 library lightgbm, catboost and XGBoost. I simply thought it would be interesting to compare how each library prioritise features differently. Now all that is left is train with the models check the feature importances. It is useful to know how many features we have and what is the contribution of each columns on average.</p><pre>print(&quot;total features:&quot;, len(fcol), &quot;, average:&quot;, 100/len(fcol))<br>=&gt; total features: 270 , average: 0.37037037037037035</pre><p>On average each feature contributed 0.37%. Let’s examine all features and their importances in percentage.</p><pre>def show_feature_importances(feature_importances):<br>    total_feature_importances = sum(feature_importances)<br>    assert len(feature_importances) == len(fcol) <em># sanity check</em><br>    for score, feature_name <strong>in</strong> sorted(zip(feature_importances, fcol), reverse=True):<br>        print(&#39;<strong>{}</strong>: <strong>{}</strong>&#39;.format(feature_name, score/total_feature_importances * 100))</pre><pre>show_feature_importances(gbm.feature_importance(importance_type=&#39;split&#39;))<br>=&gt; assetCodeT: 2.359697858792684<br>close_market_mean: 2.3284849241525687<br>volume_market_mean: 1.879018665334915<br>open/close_window_20_max: 1.4332979586740746<br>open/close_window_20_min: 1.4114489044259941<br>open/close_window_10_max: 1.3533928459953806<br>returnsOpenPrevMktres10_window_20_min: 1.33965915475373</pre><pre>... </pre><p>It’s good to see that the features I created positions nicely on top among the most important features. But it seems that many features have low importance (some even 0). Let’s pick a number (0.1% for example) that’s below average importance and use it as threshold to get all low-importance features. We can get rid of those in the next iteration or production.</p><pre>def get_non_important_features(feature_importances, threshold):<br>    total_feature_importances = sum(feature_importances)<br>    assert len(feature_importances) == len(fcol) <em># sanity check</em><br>    return [feature_name for score, feature_name <strong>in</strong> sorted(zip(feature_importances, fcol), reverse=True) if ((score * 100) / total_feature_importances)  &lt; threshold]<br><br>non_features = get_non_important_features(gbm.feature_importance(importance_type=&#39;split&#39;), threshold = 0.1)<br>print(len(non_features))<br>non_features<br>=&gt; 123<br>[&#39;open_window_5_min&#39;,<br> &#39;close/close_market_mean_window_10_median&#39;,<br> &#39;open_window_20_median&#39;,<br> &#39;close_window_5_min&#39;,</pre><pre>...</pre><p>I think this is quite good work flow in feature generation. In summary,</p><ol><li>Experiment making new features</li><li>Train models using the features</li><li>Check feature importances</li><li>Filter out unimportant features</li><li>Repeat 1–4 until you are happy with the features you have</li><li>Use those features for production / submitting</li></ol><p>Thank you so much for reading and if you find some issues in my process or have some tips, tricks &amp; suggestions, please let me know. Ah, also my Kaggle handle is <a href="https://www.kaggle.com/wontheone1">wontheone1</a>, in case you want to follow :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54669d4a7efc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/predict/feature-engineering-for-stock-price-movement-54669d4a7efc">Feature engineering for stock price movement</a> was originally published in <a href="https://medium.com/predict">Predict</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using PyTorch for Kaggle’s famous Dogs vs. Cats challenge Part 2 (inferencing and submitting)]]></title>
            <link>https://medium.com/predict/using-pytorch-for-kaggles-famous-dogs-vs-cats-challenge-part-2-inferencing-and-submitting-b5810b4dda96?source=rss-168108739f86------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5810b4dda96</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Won Seob Seo]]></dc:creator>
            <pubDate>Mon, 12 Nov 2018 15:57:50 GMT</pubDate>
            <atom:updated>2018-11-12T17:13:55.021Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*7QCHH2OVd6W46jj1v3JIvg.jpeg" /></figure><p>This story is the second part of the series where I explain how to do <a href="https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition">Dogs vs. Cats challenge on kaggle</a> using PyTorch. If you didn’t read <a href="https://medium.com/@wontheone1/using-pytorch-for-kaggles-famous-dogs-vs-cats-challenge-part-1-preprocessing-and-training-407017e1a10c">the first part</a> of the series, I recommend you read it first. It talks about data preprocessing, model training from a pretrained model, saving/loading the best model etc. In this second part, I will talk about how to make inferences on test data, writing it to csv file and submitting it to kaggle. The code shown below may need the imports, functions, variables from first part to run without errors.</p><p>So we trained our model in the previous part, now let’s first visualize how our model is doing on validation set.</p><pre>def visualize_model(model, num_images=6):<br>    was_training = model.training<br>    model.eval()<br>    images_so_far = 0<br>    fig = plt.figure()</pre><pre>    with torch.no_grad():<br>        for i, (inputs, labels) in enumerate(dataloaders[&#39;val&#39;]):<br>            inputs = inputs.to(device)<br>            labels = labels.to(device)</pre><pre>    outputs = model(inputs)<br>            _, preds = torch.max(outputs, 1)</pre><pre>            for j in range(inputs.size()[0]):<br>                images_so_far += 1<br>                ax = plt.subplot(num_images//2, 2, images_so_far)<br>                ax.axis(&#39;off&#39;)<br>                ax.set_title(&#39;predicted: {}&#39;.format(class_names[preds[j]]))<br>                imshow(inputs.cpu().data[j])</pre><pre>                if images_so_far == num_images:<br>                    model.train(mode=was_training)<br>                    return<br>        model.train(mode=was_training)</pre><pre>visualize_model(model_conv)</pre><pre>plt.ioff()<br>plt.show()</pre><p>The above code will show you 6 (by default) images from validation set and show what our model think they are. The predictions should look quite correct at this point.</p><p>To load the images with PyTorch, we need to have subfolders in the dataset directory. test folder does not have any subfolders. One way to get around this is making a dummy folder as a subfolder and put all data in the subfolder and start loading it just like the train and val image_datasets. However this is not the way I chose to do. I chose the harder way which is not depending on PyTorch for loading data. I thought this might be helpful practice in understanding how it machine learning really works behind the APIs of machine learning libraries and frameworks.</p><p>Firstly, I define a function that takes Python Image Library (PIL) image and outputs a torch.Tensor applied with same transformation as validation data.</p><pre>def apply_test_transforms(inp):<br>    out = transforms.functional.resize(inp, [224,224])<br>    out = transforms.functional.to_tensor(out)<br>    out = transforms.functional.normalize(out, [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>    return out</pre><p>To give the PIL image to this function we should first get the image as a PIL image.</p><pre>from PIL import Image</pre><pre>test_data_dir = f&#39;{data_dir}/test&#39;<br>test_data_files = os.listdir(test_data_dir)<br>im = Image.open(f&#39;{test_data_dir}/{test_data_files[0]}&#39;)<br>plt.imshow(im)</pre><p>We see that im is a PIL image that we can pass to the apply_test_transforms.</p><pre>im_as_tensor = apply_test_transforms(im)<br>print(im_as_tensor.size()) # torch.Size([3, 224, 224])<br>minibatch = torch.stack([im_as_tensor])<br>print(minibatch.size()) # torch.Size([1, 3, 224, 224])</pre><p>apply_test_transforms returns us a tensor but to feed it to PyTorch for inference, we need to convert it to a batch because PyTorch models always expect batches. Using torch.stack it is easy enough to make a single tensor into a mini batch of size 1.</p><pre>model_conv(minibatch) # tensor([[ 2.0083, -1.8386]], grad_fn=&lt;ThAddmmBackward&gt;)</pre><p>Making inference is easy! just call the model with the batch as an argument. the prediction class in this case was a cat (index 0) because 2.0083 is greater than -1.8386. But wait, the kaggle submission is expecting a csv file with id and label header and we should provide a probability that an image is a dog image for the label header. Something like the following.</p><pre>id,label<br>1,0.8<br>2,0.1<br>3,0.55<br>...</pre><p>Applying the tensor to the Softmax function will convert the tensor to the probabilities which add up to 1.</p><pre>softMax = nn.Softmax(dim = 1)<br>preds = softMax(model_conv(minibatch))<br>preds # tensor([[0.9791, 0.0209]], grad_fn=&lt;SoftmaxBackward&gt;)</pre><p>We can see that our model is 97.9% confident that the image was a cat image. Let’s define a function where we can get the dog probability by passing model and a tensor.</p><pre>def predict_dog_prob_of_single_instance(model, tensor):<br>    batch = torch.stack([tensor])<br>    softMax = nn.Softmax(dim = 1)<br>    preds = softMax(model(batch))<br>    return preds[0,1].item()</pre><p>We also need a function to get PIL image from a file name.</p><pre>def test_data_from_fname(fname):<br>    im = Image.open(f&#39;{test_data_dir}/{fname}&#39;)<br>    return apply_test_transforms(im)</pre><p>Another function we need is a function that extracts the numeric id part from file name.</p><pre>import re</pre><pre>def extract_file_id(fname):<br>    print(&quot;Extracting id from &quot; + fname)<br>    return int(re.search(&#39;\d+&#39;, fname).group())</pre><pre>extract_file_id(&quot;cat34432.jpg&quot;) # 34432</pre><p>Notice that we are returning it as an int since we want to sort ids numerically. Sorting string ids will not give us the result we want (‘12000’ is earlier than ‘39’ when sorted as string for example).</p><p>Let’s put our model to evaluation mode and start inferencing the whole test data set (12500 images). We need to switch to evaluation mode because we want to set dropouts and batch normalizations to evaluation mode. Otherwise our predictions can be inconsistent. As I am using a Macbook without GPU this step takes a lot of time.</p><pre>model_conv.eval()<br>id_to_dog_prob = {extract_file_id(fname): <br>                  predict_dog_prob_of_single_instance(model_conv,<br>                                                      test_data_from_fname(fname))<br>                  for fname in test_data_files}</pre><p>Pandas DataFrame has easy API to output to csv file so I am now going to turn id_to_dog_prob dictionary into a pandas DataFrame.</p><pre>import pandas as pd</pre><pre>ds = pd.Series({id : label for (id, label) in zip(id_to_dog_prob.keys(), id_to_dog_prob.values())})<br>ds.head()df = pd.DataFrame(ds, columns = [&#39;label&#39;]).sort_index()<br>df[&#39;id&#39;] = df.index<br>df = df[[&#39;id&#39;, &#39;label&#39;]]<br>df.head()</pre><p>Now that we see our dataframe has id and label column and everything looks as expected so let’s write to a csv file.</p><pre>df.to_csv(SUBMISSION_FILE, index = False)</pre><p>Examine the csv file and see if everything looks good. And then finally you can submit to dogs-vs-cats-redux-kernels-edition competition.</p><pre>!kaggle competitions submit -c dogs-vs-cats-redux-kernels-edition -f submission.csv -m &quot;Using PyTorch&quot;</pre><p>Even though I deleted a lot of images from training and validation datasets, still the loss was pretty low (0.07951). And it would put me at 248th position among 1314 entries in the Public Leaderboard if my solution had been submitted while the competition was still going on. Sadly, the board doesn’t seem to be updated anymore so I cannot see my kaggle handle in the board.</p><p>This wraps up the the series of 2 parts. Hopefully it was helpful to all the patient readers and feel free to tweak your approach to get even better results. Questions, claps and followings are welcome.</p><p>You can see the full code from <a href="https://github.com/wontheone1/cats-dogs-pytorch">my Github repo</a>. Everything from this part is in <a href="https://github.com/wontheone1/cats-dogs-pytorch/blob/master/catsanddogs.ipynb">catsanddogs.ipynb</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5810b4dda96" width="1" height="1" alt=""><hr><p><a href="https://medium.com/predict/using-pytorch-for-kaggles-famous-dogs-vs-cats-challenge-part-2-inferencing-and-submitting-b5810b4dda96">Using PyTorch for Kaggle’s famous Dogs vs. Cats challenge Part 2 (inferencing and submitting)</a> was originally published in <a href="https://medium.com/predict">Predict</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using PyTorch for Kaggle’s famous Dogs vs. Cats challenge Part 1 (preprocessing and training)]]></title>
            <link>https://medium.com/predict/using-pytorch-for-kaggles-famous-dogs-vs-cats-challenge-part-1-preprocessing-and-training-407017e1a10c?source=rss-168108739f86------2</link>
            <guid isPermaLink="false">https://medium.com/p/407017e1a10c</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Won Seob Seo]]></dc:creator>
            <pubDate>Mon, 12 Nov 2018 12:37:29 GMT</pubDate>
            <atom:updated>2018-11-13T09:29:32.853Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*E8m6zptHLrIh86sghKZB6w.jpeg" /></figure><p>For machine learning beginners who want to try out image classification problems, a good exercise might be building a binary classification model. <a href="https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition">Dogs vs. Cats challenge</a> is just that! Really easy concept, You just need to teach a computer to tell dogs and cats apart. One can argue this as a ‘Hello world!’ of machine learning along with MNIST. However, for complete beginners it maybe difficult to choose a good architecture, produce the output that’s in the right format for the submission etc. For that reason, I am writing this post describing what I did for this competition. I also tried to create a kaggle kernel but realized kaggle kernel is only read-only system so I cannot move my data structure around and create my submission file so I will copy past my code to the post. FYI, I did this competition on my Macbook without a single GPU.</p><p>What you can expect in this post is 1) organizing train/validation datasets, 2) transfer learning, 3) saving / loading the best model, 4) make inferences from test dataset, 5) make submission file in right format and submit to kaggle and some more. Without further a due, let’s get to it.</p><h3>1. Organizing data</h3><p>Your data comes with train data and test data. Train data has both cats and dogs but they have class in file name ( cat.&lt;id&gt;.jpg for cat images, dogs.&lt;id&gt;.jpg for dog images.). Since PyTorch support loading image data from subfolders of data directory, we will have to put all cat images to cats folder and all dog images to dogs folder. We will also have to set apart validation set to check our model is learning properly. So crate subfolders cats and dogs inside train folder, create val folder under input folder and create the same subfolders inside val folder. Test data is unlabelled and it’s ok to leave as they are.</p><pre>import os<br>train_dir = &quot;./data/train&quot;<br>train_dogs_dir = f&#39;{train_dir}/dogs&#39;<br>train_cats_dir = f&#39;{train_dir}/cats&#39;<br>val_dir = &quot;./data/val&quot;<br>val_dogs_dir = f&#39;{val_dir}/dogs&#39;<br>val_cats_dir = f&#39;{val_dir}/cats&#39;<br>print(&quot;Printing data dir&quot;)<br>print(os.listdir(&quot;data&quot;)) # Shows train, val folders are under data<br>print(&quot;Printing train dir&quot;)<br>!ls {train_dir} | head -n 5 # Shows image files are in train folder<br>print(&quot;Printing train dog dir&quot;)<br>!ls {train_dogs_dir} | head -n 5 # Check the (empty) folder exist<br>print(&quot;Printing train cat dir&quot;)<br>!ls {train_cats_dir} | head -n 5 # Check the (empty) folder exist<br>print(&quot;Printing val dir&quot;)<br>!ls {val_dir} | head -n 5  # Shows subfolder dogs and cats exist<br>print(&quot;Printing val dog dir&quot;)<br>!ls {val_dogs_dir} | head -n 5 # Check the (empty) folder exist<br>print(&quot;Printing val cat dir&quot;)<br>!ls {val_cats_dir} | head -n 5 # Check the (empty) folder exist</pre><p>Run the code above in Jupyter notebook and check we have prepared the right folder structure. Next step is move files to the right folder.</p><pre>import shutil<br>import re</pre><pre>files = os.listdir(train_dir)</pre><pre># Move all train cat images to cats folder, dog images to dogs folder<br>for f in files:<br>    catSearchObj = re.search(&quot;cat&quot;, f)<br>    dogSearchObj = re.search(&quot;dog&quot;, f)<br>    if catSearchObj:<br>        shutil.move(f&#39;{train_dir}/{f}&#39;, train_cats_dir)<br>    elif dogSearchObj:<br>        shutil.move(f&#39;{train_dir}/{f}&#39;, train_dogs_dir)</pre><p>Let’s check if we moved files correctly.</p><pre>print(&quot;Printing train dir&quot;) # shows cats, dogs subfolders only<br>!ls {train_dir} | head -n 5<br>print(&quot;Printing train dog dir&quot;) # there is now dog images in dogs folder<br>!ls {train_dogs_dir} | head -n 5<br>print(&quot;Printing train cat dir&quot;) # there is now cat images in cats folder<br>!ls {train_cats_dir} | head -n 5</pre><p>Now let’s separate some dogs images for validation set. A lot of cases you will want to separate 20% of your whole data as validation set. In this case you have 25,000 images in training set which is quite many because cats and dogs are like <a href="http://www.image-net.org/">ImageNet</a> data. I thought 20% of that are too many and it’s enough to take out 1,000 images for cats and dogs each.</p><pre>files = os.listdir(train_dogs_dir)</pre><pre>for f in files:<br>    validationDogsSearchObj = re.search(&quot;5\d\d\d&quot;, f)<br>    if validationDogsSearchObj:<br>        shutil.move(f&#39;{train_dogs_dir}/{f}&#39;, val_dogs_dir)</pre><pre>print(&quot;Printing val dog dir&quot;)<br>!ls {val_dogs_dir} | head -n 5</pre><p>Above code moves dog images whose id ranging from 5000–5999 to validation folder. And do the same for the cats image.</p><pre>files = os.listdir(train_cats_dir)</pre><pre>for f in files:<br>    validationCatsSearchObj = re.search(&quot;5\d\d\d&quot;, f)<br>    if validationCatsSearchObj:<br>        shutil.move(f&#39;{train_cats_dir}/{f}&#39;, val_cats_dir)</pre><pre>print(&quot;Printing val cat dir&quot;)<br>!ls {val_cats_dir} | head -n 5</pre><h3><strong>2. Training model</strong></h3><p>Now that data is in the right structure, it’s time to train our model. First is importing what I need for this notebook. But this is not the complete list of imports, we will import the rest as they are needed.</p><pre>import torch<br>import torch.nn as nn<br>import torch.optim as optim<br>from torch.optim import lr_scheduler<br>import numpy as np<br>import torchvision<br>from torchvision import datasets, models, transforms<br>import matplotlib.pyplot as plt<br>import time<br>import os<br>import copy<br>import math</pre><pre>print(torch.__version__)<br>plt.ion()   # interactive mode</pre><p>Let’s define training data augmentation and validation data transform.</p><pre># Data augmentation and normalization for training<br># Just normalization for validation<br>data_transforms = {<br>    &#39;train&#39;: transforms.Compose([<br>        transforms.RandomRotation(5),<br>        transforms.RandomHorizontalFlip(),<br>        transforms.RandomResizedCrop(224, scale=(0.96, 1.0), ratio=(0.95, 1.05)),<br>        transforms.ToTensor(),<br>        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>    ]),<br>    &#39;val&#39;: transforms.Compose([<br>        transforms.Resize([224,224]),<br>        transforms.ToTensor(),<br>        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>    ]),<br>}</pre><p>As data augmentation I apply a little bit of rotation, random flips and resizing + crop. The scale of resizing is 0.96–1.0. I try to avoid giving scale less than 0.96 because you can still get the wanted variation in data and there is much less risk of cutting some important part of data off (e.g. head of the cat or dog, if we don’t have head part, it will be much harder for machine to learn how each class should look like). Also a moderate ratio change should be ok for our purpose (thinner or fatter, a cat is a cat right?). About Normalization, I used some hard coded value for mean and standard deviation. These values are known to work well and used frequently. Check <a href="https://discuss.pytorch.org/t/whats-the-range-of-the-input-value-desired-to-use-pretrained-resnet152-and-vgg19/1683/2">this Facebook AI engineer’s recommendation for using those values</a> and also <a href="https://github.com/pytorch/examples/blob/42e5b996718797e45c46a25c55b031e6768f8440/imagenet/main.py#L89-L101">official PyTorch example using the same value.</a></p><pre>data_dir = &#39;data&#39;<br>CHECK_POINT_PATH = &#39;checkpoint.tar&#39;<br>SUBMISSION_FILE = &#39;submission.csv&#39;<br>image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),<br>                                          data_transforms[x])<br>                  for x in [&#39;train&#39;, &#39;val&#39;]}<br>dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,<br>                                              shuffle=True, num_workers=4)<br>              for x in [&#39;train&#39;, &#39;val&#39;]}<br>dataset_sizes = {x: len(image_datasets[x]) for x in [&#39;train&#39;, &#39;val&#39;]}<br>class_names = image_datasets[&#39;train&#39;].classes</pre><pre>device = torch.device(&quot;cuda:0&quot; if torch.cuda.is_available() else &quot;cpu&quot;)</pre><pre>print(class_names) # =&gt; [&#39;cats&#39;, &#39;dogs&#39;]<br>print(f&#39;Train image size: {dataset_sizes[&quot;train&quot;]}&#39;)<br>print(f&#39;Validation image size: {dataset_sizes[&quot;val&quot;]}&#39;)</pre><p>And then define some needed constants and define the datasets (train and val). You should see 23000 for train image size and 2000 for validation image size if you followed everything correctly. FYI, as I am working with no GPU and it frustratingly takes long time to process that much data I went to delete whole bunch of data and proceeded with 950 total train images and 71 validation images. For me that resulted in satisfactory accuracy and the purpose for me to do this was not about making most accurate model but to practice using PyTorch and kaggle website that’s why I chose to not to use so much data but of course you would not want to delete data if you are training a model for production purpose. Another reason that it was possible for me to train my model with so little data was I use pretrained model. Meaning I modified and trained only the last layer and used all the layers as they were because the model was well trained with ImageNet data already.</p><p>Let’s take a look at what a mini batch (4 images) from training set look like using the next snippet of code.</p><pre>def imshow(inp, title=None):<br>    &quot;&quot;&quot;Imshow for Tensor.&quot;&quot;&quot;<br>    inp = inp.numpy().transpose((1, 2, 0))<br>    mean = np.array([0.485, 0.456, 0.406])<br>    std = np.array([0.229, 0.224, 0.225])<br>    inp = std * inp + mean<br>    inp = np.clip(inp, 0, 1)<br>    plt.imshow(inp)<br>    if title is not None:<br>        plt.title(title)<br>    plt.pause(0.001)  # pause a bit so that plots are updated</pre><pre># Get a batch of training data<br>inputs, classes = next(iter(dataloaders[&#39;train&#39;]))</pre><pre># Make a grid from batch<br>sample_train_images = torchvision.utils.make_grid(inputs)</pre><pre>imshow(sample_train_images, title=classes)</pre><p>You will see randomly selected 4 images and title will say 0 for cat and 1 for dog. Next let’s define a function that trains our model and return some metric.</p><pre>def train_model(model, criterion, optimizer, scheduler, num_epochs=2, checkpoint = None):<br>    since = time.time()</pre><pre>if checkpoint is None:<br>        best_model_wts = copy.deepcopy(model.state_dict())<br>        best_loss = math.inf<br>        best_acc = 0.<br>    else:<br>        print(f&#39;Val loss: {checkpoint[&quot;best_val_loss&quot;]}, Val accuracy: {checkpoint[&quot;best_val_accuracy&quot;]}&#39;)<br>        model.load_state_dict(checkpoint[&#39;model_state_dict&#39;])<br>        best_model_wts = copy.deepcopy(model.state_dict())<br>        optimizer.load_state_dict(checkpoint[&#39;optimizer_state_dict&#39;])<br>        scheduler.load_state_dict(checkpoint[&#39;scheduler_state_dict&#39;])<br>        best_loss = checkpoint[&#39;best_val_loss&#39;]<br>        best_acc = checkpoint[&#39;best_val_accuracy&#39;]</pre><pre>for epoch in range(num_epochs):<br>        print(&#39;Epoch {}/{}&#39;.format(epoch, num_epochs - 1))<br>        print(&#39;-&#39; * 10)</pre><pre># Each epoch has a training and validation phase<br>        for phase in [&#39;train&#39;, &#39;val&#39;]:<br>            if phase == &#39;train&#39;:<br>                scheduler.step()<br>                model.train()  # Set model to training mode<br>            else:<br>                model.eval()   # Set model to evaluate mode</pre><pre>running_loss = 0.0<br>            running_corrects = 0</pre><pre># Iterate over data.<br>            for i, (inputs, labels) in enumerate(dataloaders[phase]):<br>                inputs = inputs.to(device)<br>                labels = labels.to(device)</pre><pre># zero the parameter gradients<br>                optimizer.zero_grad()<br>                <br>                if i % 200 == 199:<br>                    print(&#39;[%d, %d] loss: %.3f&#39; % <br>                          (epoch + 1, i, running_loss / (i * inputs.size(0))))</pre><pre># forward<br>                # track history if only in train<br>                with torch.set_grad_enabled(phase == &#39;train&#39;):<br>                    outputs = model(inputs)<br>                    _, preds = torch.max(outputs, 1)<br>                    loss = criterion(outputs, labels)</pre><pre># backward + optimize only if in training phase<br>                    if phase == &#39;train&#39;:<br>                        loss.backward()<br>                        optimizer.step()</pre><pre># statistics<br>                running_loss += loss.item() * inputs.size(0)<br>                running_corrects += torch.sum(preds == labels.data)</pre><pre>epoch_loss = running_loss / dataset_sizes[phase]<br>            epoch_acc = running_corrects.double() / dataset_sizes[phase]</pre><pre>print(&#39;{} Loss: {:.4f} Acc: {:.4f}&#39;.format(<br>                phase, epoch_loss, epoch_acc))</pre><pre># deep copy the model<br>            if phase == &#39;val&#39; and epoch_loss &lt; best_loss:<br>                print(f&#39;New best model found!&#39;)<br>                print(f&#39;New record loss: {epoch_loss}, previous record loss: {best_loss}&#39;)<br>                best_loss = epoch_loss<br>                best_acc = epoch_acc<br>                best_model_wts = copy.deepcopy(model.state_dict())</pre><pre>print()</pre><pre>time_elapsed = time.time() - since<br>    print(&#39;Training complete in {:.0f}m {:.0f}s&#39;.format(<br>        time_elapsed // 60, time_elapsed % 60))<br>    print(&#39;Best val Acc: {:.4f} Best val loss: {:.4f}&#39;.format(best_acc, best_loss))</pre><pre># load best model weights<br>    model.load_state_dict(best_model_wts)<br>    return model, best_loss, best_acc</pre><p>The function first checks if saved checkpoint is passed. If yes, then it will load the saved parameter and start training from where it left off. If no, then it starts training the model it was passed (we will still use pretrained model from beginning). The function updates parameters only in train phase and prints out some metrics every epochs or whenever it has new best loss.</p><p>Now let’s define our model by downloading pretrained model. It takes some time to run next code if you never ran it before.</p><pre>model_conv = torchvision.models.resnet50(pretrained=True)</pre><p>resnet50 is a convolutional neural network architecture that is really powerful for solving computer vision problems. Less powerful but less resource consuming models you might be interested in using include resnet18 and resnet34. By giving pretrained=True as an argument you will download a model with the parameters trained with ImageNet data set. Since we need to change the model for our needs (binary class classification), we will change the last fully connected layer and define a loss function that’s useful for classification problem (cross entropy loss, which combines log softmax and negative log likelihood loss function). Optimzier is stochastic gradient descent optimizer and scheduler is exponential because it will reduce learning rate by the factor of 10 every 7 epochs (In reality I only trained 6 epochs).</p><pre>for param in model_conv.parameters():<br>    param.requires_grad = False</pre><pre># Parameters of newly constructed modules have requires_grad=True by default<br>num_ftrs = model_conv.fc.in_features<br>model_conv.fc = nn.Linear(num_ftrs, 2)</pre><pre>model_conv = model_conv.to(device)</pre><pre>criterion = nn.CrossEntropyLoss()</pre><pre># Observe that only parameters of final layer are being optimized<br>optimizer_conv = optim.SGD(model_conv.fc.parameters(), lr=0.001, momentum=0.9)</pre><pre># Decay LR by a factor of 0.1 every 7 epochs<br>exp_lr_scheduler = lr_scheduler.StepLR(optimizer_conv, step_size=7, gamma=0.1)</pre><p>We can finally get into actual training.</p><pre>try:<br>    checkpoint = torch.load(CHECK_POINT_PATH)<br>    print(&quot;checkpoint loaded&quot;)<br>except:<br>    checkpoint = None<br>    print(&quot;checkpoint not found&quot;)</pre><pre>model_conv, best_val_loss, best_val_acc = train_model(model_conv,<br>                                                      criterion,<br>                                                      optimizer_conv,<br>                                                      exp_lr_scheduler,<br>                                                      num_epochs = 3,<br>                                                      checkpoint = checkpoint)</pre><pre>torch.save({&#39;model_state_dict&#39;: model_conv.state_dict(),<br>            &#39;optimizer_state_dict&#39;: optimizer_conv.state_dict(),<br>            &#39;best_val_loss&#39;: best_val_loss,<br>            &#39;best_val_accuracy&#39;: best_val_acc,<br>            &#39;scheduler_state_dict&#39; : exp_lr_scheduler.state_dict(),<br>            }, CHECK_POINT_PATH)</pre><p>The code first checks if any checkpoint is saved from previous training. If yes, pass the checkpoint to trainmodel function. Here I specified 3 epochs and the function returns model, loss, accuracy of when the loss was lowest during all epochs. We save what we got from the function to a checkpoint. You can adjust epoch number or rerun this snippet as many as you like. If you see the model is not improving anymore you can stop. As I had only 71 validation set images I reached accuracy 1.0 with loss 0.036 in only 2 runs so I decided to stop training.</p><p>Notice that we only needed to train the last layer we changed from original resnet50. It’s possible we also train all the parameters in all the layer but when I tried that, I only saw loss and accuracy getting worse. If you want to try updating all parameters you can do so like the following.</p><pre>for param in model_conv.parameters():<br>    param.requires_grad = True</pre><pre>model_conv = model_conv.to(device)</pre><pre># Observe that all parameters are being optimized<br>optimizer_ft = optim.SGD(model_conv.parameters(), lr=0.001, momentum=0.9)</pre><p>And then run the same training loop code as we did before (the code block starts with try ).</p><p>Maybe this story is getting too lengthy already. So I will write the part you actually make inferences with the model using test data set and submitting your answer to kaggle in a separate story. Tune in for the next part and some more. Let me know if you succeeded in following my code and also if you got the satisfactory result using this method.</p><p>The second (last) part is out. If you want to continue to read, <a href="https://medium.com/@wontheone1/using-pytorch-for-kaggles-famous-dogs-vs-cats-challenge-part-2-inferencing-and-submitting-b5810b4dda96">click here</a>. You can also see the full code in <a href="https://github.com/wontheone1/cats-dogs-pytorch">my Github repo</a>. Code for data preprocessing can be found in <a href="https://github.com/wontheone1/cats-dogs-pytorch/blob/master/datapreprocessor.ipynb">datapreprocessor.ipynb</a> and training, inferences and submitting can be found in <a href="https://github.com/wontheone1/cats-dogs-pytorch/blob/master/catsanddogs.ipynb">catsanddogs.ipynb</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=407017e1a10c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/predict/using-pytorch-for-kaggles-famous-dogs-vs-cats-challenge-part-1-preprocessing-and-training-407017e1a10c">Using PyTorch for Kaggle’s famous Dogs vs. Cats challenge Part 1 (preprocessing and training)</a> was originally published in <a href="https://medium.com/predict">Predict</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using fastai on Google Cloud Platform]]></title>
            <link>https://medium.com/predict/using-fastai-on-google-cloud-platform-46d3decd0da8?source=rss-168108739f86------2</link>
            <guid isPermaLink="false">https://medium.com/p/46d3decd0da8</guid>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[google-compute-engine]]></category>
            <category><![CDATA[fastai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Won Seob Seo]]></dc:creator>
            <pubDate>Mon, 08 Oct 2018 20:02:46 GMT</pubDate>
            <atom:updated>2018-10-13T20:00:57.329Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting up fastai on Google Cloud Platform</h3><p>I recently followed along a well-written medium story about <a href="https://medium.com/@howkhang/ultimate-guide-to-setting-up-a-google-cloud-machine-for-fast-ai-version-2-f374208be43">setting up Google Cloud machine for fast.ai</a>. While it is a very detailed and nice guide, it worked for me once but it does not work anymore for me so I think the script used is broken and needs update. So I will try not to repeat what’s already been said, but will try to update or suggest slightly different ways of doing things. So I suggest first reading the original guide and then come back here to find out what I suggest differently.</p><p>At the point in the guide you have a compute engine running and supposed to run the following after connecting with ssh,</p><pre>curl https://raw.githubusercontent.com/howkhang/fastai-v2-setup/master/setup.sh | bash</pre><p>Please don’t run the above and continue reading. Currently starting jupyter notebook after running above script fails for me (let me know if you succeeds in working though). So I updated the script. Run the following in the terminal once you ssh into your GCP machine.</p><pre>curl https://raw.githubusercontent.com/wontheone1/fast-ai-setup-on-gcp/e28519a5b592dc942879d3dc2a1851aac806c227/setup.sh | bash -x</pre><p>And then ssh connection will drop automatically as the script runs sudo reboot. Now the setup is finished, you can ssh into it again and start notebook by jupyter notebook. Open another terminal and run</p><pre>gcloud compute ssh pghoou@fast-ai --ssh-flag=&quot;-L&quot; --ssh-flag=&quot;8888:localhost:8888&quot;</pre><p>Then you can use the notebook on your localhost:8888 on your browser. Easy! isn’t it? If you only care about setting up and start using jupyter notebook then there’s nothing more for you to read but if you are interested then I will explain what I changed from original script and why.</p><p>The first difference I have from original script is following.</p><p>Original:</p><pre>echo &quot;c.NotebookApp.ip = &#39;*&#39;&quot; &gt;&gt; ~/.jupyter/jupyter_notebook_config.py</pre><p>Mine:</p><pre>echo &quot;c.NotebookApp.ip = &#39;0.0.0.0&#39;&quot; &gt;&gt; ~/.jupyter/jupyter_notebook_config.py</pre><p>That line of original script was the reason why install was failing for me and I learned from <a href="http://forums.fast.ai/t/may-have-fix-for-your-jupyter-problem/24567?u=wontheone1">fastai forum</a> that by changing * to 0.0.0.0 notebook will work again. And then the other change I made to original script is the next part</p><pre>wget https://repo.continuum.io/archive/Anaconda3-5.0.1-Linux-x86_64.sh<br>bash Anaconda3-5.0.1-Linux-x86_64.sh -b</pre><p>Since the latest version of Anaconda I can find is 5.3.0 and I wanted to delete the installation file once I finished installing Anaconda I changed the original script to following.</p><pre># INSTALL ANACONDA 5.3.0 </pre><pre>ANACONDA=Anaconda3-5.3.0-Linux-x86_64.sh </pre><pre>wget <a href="https://repo.continuum.io/archive/$ANACONDA">https://repo.continuum.io/archive/$ANACONDA</a><br>bash $ANACONDA -b<br>rm $ANACONDA</pre><p>Also some trivial things I changed from original script is that a) my script does NOT download the dogs and cats dataset unlike the original one. That’s because I thought not everyone who sets up fastai on GCP wants to do dogs and cats problem so I wanted this script to serve more general population.</p><p>Next part I replaced is the following,</p><pre>echo &#39;export PATH=~/anaconda3/bin:$PATH&#39; &gt;&gt; ~/.bashrc<br>export PATH=~/anaconda3/bin:$PATH<br>source ~/.bashrc</pre><p>and my script is</p><pre>sudo ln -s /home/$USER/anaconda3/etc/profile.d/conda.sh /etc/profile.d/conda.sh</pre><p>The reason for this change is adding Anaconda bin directory to PATH env variable is not a recommended way of making conda command available. Refer<a href="https://github.com/conda/conda-docs/issues/566"> this Github issue</a> and <a href="https://github.com/conda/conda/blob/master/CHANGELOG.md#440-2017-12-20">change log in conda repository</a>.</p><p>Most of other parts remain the same as in original script but small trivial things I added are</p><pre># enable nvcc --version command<br>sudo apt install nvidia-cuda-toolkit -y</pre><pre># Kaggle cli<br>pip install --upgrade pip<br>pip install kaggle</pre><p>These are just the things I find myself keep using when I setup a machine for machine learning and I think most of other people will find it useful as well. Last but not least, I use bash’s -x option to printout the all the commands in the script. I found it useful when modifying the setup script. I recommend you to also keep it that way since this kinda of script can be broken in the future since it depends too much on external world. For example, links can become unavailable or fastai’s future version may require different kinds of setup. So in case it breaks in the future, using -x option will be useful in debugging the script and updating it.</p><p>In case you want to download the cats and dogs data set like the original, you can run the following on your terminal. Doing so will also create a symlink for ~/data in ~/fastai/courses/dl1.</p><pre>cd ~<br>mkdir data<br>cd data<br>wget http://files.fast.ai/data/dogscats.zip<br>unzip -q dogscats.zip<br>cd ~/fastai/courses/dl1/<br>ln -s ~/data ./</pre><p>That’s all for explaining my script. I also want to talk about small tip on GCP. I needed to create and delete VM instance multiple times to test my script. And you could be creating / deleting VM multiple times for whatever reasons as well. What I recommend in those situation is not just create VM every time you have to. Because whenever you do it, you have to choose OS, number of CPU, memory, storage, network tag etc. Instead of doing that, create an Instance templates once and whenever you need to create a VM you can create it from the template so no need to choose all the options every time you do so.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H-wweo7zNtatjrznoW7djQ.png" /><figcaption>Compute Engine &gt; Instance templates</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*re8DLdWlxIHCGFriz4trxA.png" /><figcaption>An example of an template, you can see the ‘CREATE VM’ button</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*FlP6WzSg9G9gln2j3upkqQ.png" /><figcaption>Creating an instance from a template</figcaption></figure><p>To create an instance template, go to ‘Compute Engine &gt; Instance templates’. Click ‘CREATE INSTANCE TEMPLATE’. Choose everything as you would when you create a single VM instance. Click the name of the template you created, Click ‘CREATE VM’. Decide name and click ‘CREATE’. That is it for creating instances from template.</p><p>I hope this was helpful to you, share with me if you have a better setup process.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46d3decd0da8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/predict/using-fastai-on-google-cloud-platform-46d3decd0da8">Using fastai on Google Cloud Platform</a> was originally published in <a href="https://medium.com/predict">Predict</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Google Colab for MNIST with fastai v1]]></title>
            <link>https://medium.com/predict/using-google-colab-for-mnist-with-fastai-v1-792381bab7da?source=rss-168108739f86------2</link>
            <guid isPermaLink="false">https://medium.com/p/792381bab7da</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[mnist]]></category>
            <category><![CDATA[fastai]]></category>
            <category><![CDATA[colab]]></category>
            <dc:creator><![CDATA[Won Seob Seo]]></dc:creator>
            <pubDate>Mon, 08 Oct 2018 12:02:24 GMT</pubDate>
            <atom:updated>2018-10-13T20:04:04.787Z</atom:updated>
            <content:encoded><![CDATA[<p>Now that fastai version 1 is out, I decided to get my hands dirty. On Google Colab, you can use GPU for free for your notebook project. So I picked Colab to try out all new fastai version 1 (1.0.5 to be exact).</p><p>You can create an ipynb on your Gdrive by `new&gt;more&gt;Colaboratory`.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3OJstDOyJxtYL4NFCfjl7Q.png" /></figure><p>Once you open the new ipynb file, you can change your runtime to use GPU by going to `Runtime &gt; Change runtime type`. Select Python3 and GPU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SM3FbpXt_QHtjJOPS6rnWA.png" /></figure><p>By the way, I think Colab is ok to try out machine learning with small data but it’s not enough for serious machine learning. (Probably because it is free,) many times you will encounter GPU not available error. Then you would have to try later or use CPU environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TLXnB4OO7MyaDeFsO6HmyQ.png" /></figure><p>I found handy code from a [Github page](<a href="https://github.com/chungbrain/Google-Colab-with-GPU">https://github.com/chungbrain/Google-Colab-with-GPU</a>) that I adapted. What it does is, it installs PyTorch preview (nightly) build which fastai v1 depends on. It checks if hardware has Nvidia GPU installed and installs CPU only build if GPU is not available. If everything goes as expected, you should see nightly build version printed (something similar to `1.0.0.dev20181005`) and True from both torch.cuda.is_available() and torch.backends.cudnn.enabled .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MSnYbiYNoAYXcaooidFPHw.png" /></figure><p>And then you install fastai. The library is in active development, so I specified a version to be “1.0.5” in case there maybe a breaking change in API in the future.</p><p>Now I will go into the MNIST example which I adapted from official fastai Github(<a href="https://github.com/fastai/fastai/blob/master/examples/vision.ipynb">https://github.com/fastai/fastai/blob/master/examples/vision.ipynb</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RoSbWBMT8jK_Umoaj1hliA.png" /></figure><p>These are all imports you need to follow this short tutorial. Library author Jeremy promotes importing by * for faster iterations and better interactivity. I think using * is good for fast prototyping and experiments also. But importing specifically one by one should clarify where everything is coming from so it should help readers of this post and me to better learn about this library. Also, I am not experimenting with this notebook anymore now that I am writing about it so there is no reason for me to use * import. If I continue using this style of importing in the future posts, now you know why I do so (also I come from Software engineering background where specific import s are much appreciated). But feel free to use * import.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*VvJ-H-OBKt4_yN1YikKowg.png" /></figure><p>`MNIST_PATH` is path to MNIST data which is provided by fastai.docs . And we untar(decompress) the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*rIr96nPy1f7BqQfiKqTSgQ.png" /></figure><p>We can MNIST_PATH contains train, valid(ation) directories and a label csv file. And each of the directories has subdirectory 3 and 7 . Fully-fledged MNIST would have all 10 digits but I guess this is enough for just demonstrating image classification example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*bU8IlQsaVvxszNDHniCEUA.png" /></figure><p>Examination of contents inside of the csv file shows that there are name column and then label column. Name has path to the individual examples and label seems to have value 0 and 1. From what we can see so far we can guess 0 label means the image file has number 3 and 1 means the image file has number 7.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y_5gbEoKgudBGxz4zE478Q.png" /></figure><p>Now we load the data by passing the path to data and data set transformation function and we get the data object. Here we only apply some random paddings for data augmentation purpose. If we index to the 0th item in the train data set, we get image and label. And the image has number 3 in it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H0XGeTweps_l2bwVQK1fSA.png" /></figure><p>Now we pass data, architecture (resnet18 in this case) and metrics to ConvLearner (Conv as in Convolutional, we will use convolutional neural network because this is computer vision problem) and we get back the learn object. We then train 1 epoch by calling fit on the object. Measuring accuracy gives me that it is 98.6% accurate in distinguishing 3 from 7. Well, for many many good reasons you would probably want to use [log loss](<a href="http://wiki.fast.ai/index.php/Log_Loss">http://wiki.fast.ai/index.php/Log_Loss</a>) as your metrics but let’s just leave it this way for this post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*oNLvPQo6bOJS6EWd3VnqGQ.png" /></figure><p>Lastly, what do we actually get by calling get_preds method? It turns out that it returns a) predictions as probabilities of an example being a certain class and b) actual labels that we can use predictions against to measure the loss.</p><p>Well that was it for this post. Please give me feed back so that I can improve my post. I don’t think I will use Colab that much for machine learning purpose , but it seems like it’s good deal for the bucks (which is 0) and great way to collaborate online using notebook like environment. The fact that it needs almost no setup (installing Anacoda or such for the notebook) is also very good. I will continue writing about machine learning but I will move on to Google Cloud Platform Compute engine the next time. It gives 300$ worth of free credits for trial so I think it is a great option to practice machine learning with arbitrarily powerful machine for free (well at least in the beginning). Hope this helped.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=792381bab7da" width="1" height="1" alt=""><hr><p><a href="https://medium.com/predict/using-google-colab-for-mnist-with-fastai-v1-792381bab7da">Using Google Colab for MNIST with fastai v1</a> was originally published in <a href="https://medium.com/predict">Predict</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine learning journey with Won]]></title>
            <link>https://medium.com/series/machine-learning-journey-with-won-deca817b0593?source=rss-168108739f86------2</link>
            <guid isPermaLink="false">https://medium.com/p/deca817b0593</guid>
            <dc:creator><![CDATA[Won Seob Seo]]></dc:creator>
            <pubDate>Sun, 07 Oct 2018 18:31:41 GMT</pubDate>
            <atom:updated>2018-10-07T18:31:41.686Z</atom:updated>
            <content:encoded><![CDATA[<p>Using Google Colab for MNIST with fastai v1</p><p>Now that fastai version 1 is out, I decided to get my hands dirty on it. On Google Colab, you can use GPU for free for your notebook project. So I picked Colab to try out all new fastai version 1 (1.0.5 to be exact).</p><p>You can create an ipynb on your Gdrive by `new&gt;more&gt;Colaboratory`.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3OJstDOyJxtYL4NFCfjl7Q.png" /></figure><p>Once you open the new ipynb file, you can change your runtime to use GPU by going to `Runtime &gt; Change runtime type`. Select Python3 and GPU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SM3FbpXt_QHtjJOPS6rnWA.png" /></figure><p>By the way, I think Colab is ok to try out machine learning with small data but it’s not enough for serious machine learning. (Probably because it is free,) many times you will encounter GPU not available error. Then you would have to try later or use CPU environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TLXnB4OO7MyaDeFsO6HmyQ.png" /></figure><p>I found handy code from a [Github page](<a href="https://github.com/chungbrain/Google-Colab-with-GPU">https://github.com/chungbrain/Google-Colab-with-GPU</a>) that I adapted. What it does is, it installs PyTorch preview (nightly) build which fastai v1 depends on. It checks if hardware has Nvidia GPU installed and installs CPU only build if GPU is not available. If everything goes as expected, you should see nightly build version printed (something similar to `1.0.0.dev20181005`) and True from both `torch.cuda.is_available()` and `torch.backends.cudnn.enabled`</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MSnYbiYNoAYXcaooidFPHw.png" /></figure><p>And then you install fastai. The library is in active development, so I specified a version to be “1.0.5” in case there maybe a breaking change in API.</p><p>Now I will go into the MNIST example which I adapted from official fastai Github(<a href="https://github.com/fastai/fastai/blob/master/examples/vision.ipynb">https://github.com/fastai/fastai/blob/master/examples/vision.ipynb</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RoSbWBMT8jK_Umoaj1hliA.png" /></figure><p>These are all imports you need to follow this short tutorial. Library author Jeremy promotes importing by * for faster iterations and better interactivity. I think using * is good for fast prototyping and experiments also. But importing specifically one by one should clarify where everything is coming from so it should help readers of this post and me to better learn about this library. Also, I am not experimenting with this notebook anymore now that I am writing about it so there is no reasons to use * import. If I continue this style of importing in the future posts, then you now know why (also I come from Software engineering background where specific imports are much appreciated).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*VvJ-H-OBKt4_yN1YikKowg.png" /></figure><p>`MNIST_PATH` is path to MNIST data which is provided by fastai.docs . And we untar(decompress) the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*rIr96nPy1f7BqQfiKqTSgQ.png" /></figure><p>We can MNIST_PATH contains train, valid(ation) directories and a label csv file. And each of the directories has subdirectory 3 and 7 . Fully-fledged MNIST would have all 10 digits but I guess this is enough for just demonstrating image classification example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*bU8IlQsaVvxszNDHniCEUA.png" /></figure><p>Examination of contents inside of the csv file shows that there are name column and then label column. Name has path to the individual examples and label seems to have value 0 and 1. From what we can see so far we can guess 0 label means the image file has number 3 and 1 means the image file has number 7.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y_5gbEoKgudBGxz4zE478Q.png" /></figure><p>Now we load the data by passing the path to data and data set transformation function and we get the data object. Here we only apply some random paddings for data augmentation purpose. If we index to the 0th item in the train data set, we get image and label. And the image has number 3 in it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H0XGeTweps_l2bwVQK1fSA.png" /></figure><p>Now we pass data, architecture (resnet18 in this case) and metrics to ConvLearner (Conv as in Convolutional, we will use convolutional neural network because this is computer vision problem) and we get back the learn object. We then train 1 epoch by calling fit on the object. Measuring accuracy gives me that it is 98.6% accurate in distinguishing 3 from 7. Well, for many many good reasons you would probably want to use [log loss](<a href="http://wiki.fast.ai/index.php/Log_Loss">http://wiki.fast.ai/index.php/Log_Loss</a>) as your metrics but let’s just leave it this way for this post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*oNLvPQo6bOJS6EWd3VnqGQ.png" /></figure><p>Lastly, what do we actually get by calling get_preds method? It turns out that it returns 1. predictions as probabilities of an example being a certain class and 2. actual labels that we can use predictions against to measure the loss.</p><p>Well that was it for this post. Please give me feed back so that I can improve my post. I don’t think I will use Colab that much for machine learning purpose but it seems like it’s great deal for the bucks (which is 0) and great way to collaborate online using notebook like environment. The fact that it needs almost no setup is also very strong point. I want to continue writing about machine learning but I will touch on Google Cloud Platform Compute engine the next time. It gives 300$ worth of free credits for trial so I think it is a great option to practice machine learning with arbitrarily powerful machine for no cost (well at least for the beginning). Hope this helped! Please tune in for the future posts!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=deca817b0593" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>