<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Neel Iyer on Medium]]></title>
        <description><![CDATA[Stories by Neel Iyer on Medium]]></description>
        <link>https://medium.com/@neel-iyer?source=rss-e7c5e1c9cae1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xuZK6oUtg9fGBS79Gms6ew.png</url>
            <title>Stories by Neel Iyer on Medium</title>
            <link>https://medium.com/@neel-iyer?source=rss-e7c5e1c9cae1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 00:48:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@neel-iyer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I Got Jacked in 12 Months]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://neel-iyer.medium.com/how-i-got-jacked-in-12-months-cdf103bf3a1f?source=rss-e7c5e1c9cae1------2"><img src="https://cdn-images-1.medium.com/max/1080/1*i60dELB3vK1-WzhnNeXn1g.png" width="1080"></a></p><p class="medium-feed-snippet">You Never See A Fit Indian IRL</p><p class="medium-feed-link"><a href="https://neel-iyer.medium.com/how-i-got-jacked-in-12-months-cdf103bf3a1f?source=rss-e7c5e1c9cae1------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://neel-iyer.medium.com/how-i-got-jacked-in-12-months-cdf103bf3a1f?source=rss-e7c5e1c9cae1------2</link>
            <guid isPermaLink="false">https://medium.com/p/cdf103bf3a1f</guid>
            <category><![CDATA[weightloss-foods]]></category>
            <category><![CDATA[belly-fat]]></category>
            <category><![CDATA[fitness]]></category>
            <category><![CDATA[weight-loss]]></category>
            <dc:creator><![CDATA[Neel Iyer]]></dc:creator>
            <pubDate>Thu, 30 Nov 2023 13:03:01 GMT</pubDate>
            <atom:updated>2023-11-30T13:03:01.185Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How I lost all my money in Crypto]]></title>
            <link>https://medium.com/swlh/how-i-lost-all-my-money-in-crypto-f39657baffd7?source=rss-e7c5e1c9cae1------2</link>
            <guid isPermaLink="false">https://medium.com/p/f39657baffd7</guid>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[money]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[ethereum]]></category>
            <dc:creator><![CDATA[Neel Iyer]]></dc:creator>
            <pubDate>Mon, 05 Apr 2021 11:53:03 GMT</pubDate>
            <atom:updated>2022-09-27T23:30:08.975Z</atom:updated>
            <content:encoded><![CDATA[<h4>Cocky, overconfident and stupid</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ax71lbYD71hdyef0CmT3Bw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@harrisonkugler?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Harrison Kugler</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In 2017, I invested everything I had into Crypto. I didn’t have much. But I was all in. As you can probably guess from the title, things didn’t go so well.</p><p>For the first couple of months I was ecstatic. Prices kept going up. I was making more money in 24 hours that I had ever made in a regular job.</p><p>I began watching crypto youtubers “explain” how various coins worked. I bought into the hype completely.</p><p>I told my friends to buy Crypto. “You don’t want to tell your grandkids you missed out, do you?”</p><p>I’d go onto coinmarketcap several times a day and check the price. Was BTC up in the past 24 hours? Was ETH up in the past 24 hours? What about my Alts? Should I buy the dip?</p><p>I threw all caution to the wind.</p><p>I began lecturing people on how the blockchain works. I didn’t understand what I was talking about. And yet I would tell people how the distributed ledger system would revolutionise the monetary system.</p><p>Somehow I had convinced myself that I knew what I was talking about.</p><p>Then it all went to shit. The entire crypto market fell. Sharply first. Then it stabilised for a while. Then it fell faster and faster.</p><p>I distinctly remembering logging into my porfolio tracker at one point: I had lost nearly $7000 in 24 hours. That is so much money.</p><p>Aside from that number, I don’t remember exact figures and stats. It was all a blur. I remember the prevailing color of red. I would wake up, check my portfolio and see a sea of red.</p><p>But for some reason I didn’t sell. I definitely panicked. But I didn’t sell. I continued to HODL.</p><p>In my mind I thought I was being smart. But in reality I didn’t want to be proven wrong. By holding on I still had hope. I told myself: “Maybe a year or two from now I’ll be back in the green. Then I’ll show them.”</p><p>I didn’t really know who ‘they’ were but I was going to show them anyway.</p><p>A year later I sold.</p><p>I had lost more than my original investment. But more than that I was very publicly wrong about something. I felt humiliated. I got angry when people would bring up crypto in conversation. I’d desperately try to change the subject.</p><p>The market had taught me a valuable lesson. But at the time I was too stubborn to see it.</p><p>The reason I write about this is because I think there may be a reckoning coming to Crypto currencies soon. I’m still a believer in them long-term, but I don’t want to see anyone get burned in the same way I did. Please don’t delude yourself.</p><p>Very few people really understand what’s going on here. This is still a very nascent space.</p><p>You might think you understand ETH, but do you really? Could you explain it to a 5 year old? I’m not judging here. I’ve been researching this for a long time and I still don’t think I get it.</p><p>I think Naval puts it best. He said something along the lines of this shouldn’t be more than 2% of your portfolio. And you should try to invest through funds. Holding the crypto yourself is very risky. <a href="https://www.techradar.com/au/news/yet-another-crypto-exchange-has-fallen-victim-to-a-massive-hack#:~:text=A%20cryptocurrency%20exchange%20with%20a,stolen%20from%20its%20hot%20wallets.&amp;text=In%20addition%2C%20EXMO%20has%20reiterated,this%20incident%20will%20be%20refunded.">Exchanges get hacked</a>. Online wallets aren’t that <a href="https://techcrunch.com/2021/03/16/5-7m-stolen-in-roll-crypto-heist-after-hot-wallet-hacked/">safe</a>. ̶E̶v̶e̶n̶ ̶s̶u̶p̶p̶o̶s̶e̶d̶l̶y̶ ̶”̶s̶a̶f̶e̶”̶ ̶s̶o̶l̶u̶t̶i̶o̶n̶s̶ ̶l̶i̶k̶e̶ ̶<a href="https://www.forbes.com/sites/kenrapoza/2021/12/28/after-ledger-hack-who-can-you-trust-for-bitcoin-storage/">L̶e̶d̶g̶e̶r̶ ̶X̶ ̶g̶e̶t̶ ̶h̶a̶c̶k̶e̶d̶.̶</a></p><p>Look, I’m not here to tell you what to do with your money (and this is not financial advice). Just please don’t delude yourself in the same way I did. Because after all:</p><blockquote>“The first principle is that you must not fool yourself and you are the easiest person to fool.” — Richard Feynman</blockquote><p>If you want to support me consider signing up bybit (my preferred exchange) here: <a href="https://www.bybit.com/en-US/invite?ref=5W8QVK">https://www.bybit.com/en-US/invite?ref=5W8QVK</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f39657baffd7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/how-i-lost-all-my-money-in-crypto-f39657baffd7">How I lost all my money in Crypto</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Segment Satellite Imagery using NDVI and Rasterio]]></title>
            <link>https://medium.com/data-science/segment-satellite-imagery-using-ndvi-and-rasterio-6dcae02a044b?source=rss-e7c5e1c9cae1------2</link>
            <guid isPermaLink="false">https://medium.com/p/6dcae02a044b</guid>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[satellite]]></category>
            <category><![CDATA[satellite-imagery]]></category>
            <category><![CDATA[remote-sensing]]></category>
            <dc:creator><![CDATA[Neel Iyer]]></dc:creator>
            <pubDate>Sun, 31 Jan 2021 00:00:00 GMT</pubDate>
            <atom:updated>2021-02-02T21:03:11.113Z</atom:updated>
            <content:encoded><![CDATA[<h4>Easily Segment Canopy Cover and Soil using NDVI and Rasterio</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nsu8a-KrT_4sKaXI.png" /><figcaption>Image by Author</figcaption></figure><p>In this post we’ll be trying to segment canopy cover and soil in satellite imagery.</p><p>We’ll be borrowing ideas from <a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0196605">this paper</a>. I’ll also be using ideas from my <a href="https://towardsdatascience.com/segment-satellite-images-using-rasterio-and-scikit-learn-fc048f465874">previous</a> blog post on this topic.</p><p>Ideally we want to go from a regular satellite image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/0*s-SZFEc5MEGUrHlQ.png" /><figcaption>Image by Author</figcaption></figure><p>To this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/419/0*8rTjXw2izHbmbAls.png" /><figcaption>Image by Author</figcaption></figure><p>The orange is soil. The red is vegetation.</p><h3>NDVI</h3><p>As the paper notes we’ll need to extract the Normalized difference vegetation index. This is a useful index for vegetation. Here’s an example of a NDVI image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/0*Q1A_O1R8fJWz9i6a.png" /><figcaption>Image by Author</figcaption></figure><p>And <a href="https://www.researchgate.net/publication/342413913/figure/fig2/AS:905930921226240@1593002171931/Formula-used-to-calculate-the-normalized-difference-vegetation-index-NDVI.ppm">here’s</a> the formula</p><p>Here’s my code to obtain the NDVI image as a numpy array.</p><pre>def get_ndvi(red_file):<br><br>    nir_file = os.path.dirname(red_file) + &#39;/nir.tif&#39;<br>    band_red = rasterio.open(red_file)<br>    band_nir = rasterio.open(nir_sfile)<br>    red = band_red.read(1).astype(&#39;float64&#39;)<br>    nir = band_nir.read(1).astype(&#39;float64&#39;)<br><br>    # ndvi calculation, empty cells or nodata cells are reported as 0<br>    ndvi=np.where( (nir==0.) | (red ==0.), -255 , np.where((nir+red)==0., 0, (nir-red)/(nir+red)))<br><br>    return ndvi</pre><p>The NDVI is calculated for a certain region of interest. This is defined in the paper:</p><blockquote>For each field plot, a region of interest (ROI) was established manually by choosing the central two rows and mean value of vegetation indices were extracted corresponding to each plot.</blockquote><p>This is pretty simple to implement in python.</p><pre># get average ndvi of center two rows<br>  def get_region_of_interest(ndvi, multiplier = 1/2):<br><br>    # undo the background adjustment<br>    region = ndvi.copy()<br>    region = np.where(region==-255, 0, region)<br><br>    # mean of center rows<br>    center_row1 = np.mean(region[int((multiplier)*len(region))])<br>    center_row2 = np.mean(region[int((multiplier)*len(region))+1])<br><br>    # mean of both rows<br>    mean = (center_row1.copy()+center_row2.copy())/2<br><br>    return mean</pre><h3>Fractional Vegetation Cover</h3><p>The paper also makes reference to fractional vegetation cover. I’ll calculate this by setting a threshold. Any NDVI value above that threshold will be vegetation. Anything below that will be soil.</p><pre>THRESHOLD = 0.3<br><br>def get_fc(ndvi):<br><br>    ndvi_copy = ndvi.copy()<br><br>    vegetation = np.where(ndvi_copy &gt; THRESHOLD, 1, 0)<br>    vegetation_count = np.count_nonzero(vegetation)<br><br>    total = ndvi_copy.shape[0]*ndvi_copy.shape[1]<br>    fractional_cover = vegetation_count/total<br><br>    return fractional_cover</pre><p>We’ll need to change that threshold value later on.</p><h3>Plot Fractional Vegetation Cover vs NDVI</h3><p>Now we can recreate the plots on page 7 of the paper. We’ll be plotting fractional vegeation cover vs NDVI for each image.</p><p>We also want to plot the least squares regression line.</p><p>Here’s a snippet of code that allows us to do that.</p><pre>def plot_fc_vs_ndvi(fc, ndvi):<br><br>    y = np.array(fc).reshape(1, -1)<br>    x = np.array(ndvi).reshape(1,-1)<br><br>    slope, intercept, r_value, p_value, std_err = stats.linregress(x,y)<br><br>    x = np.linspace(min(ndvi),max(ndvi),100)<br>    f, ax = plt.subplots(1,1,figsize=(10,10))<br>    ax.plot(x, slope*x+intercept, &#39;-r&#39;, label=&#39;fc=&#39;+str(round(slope, 2))+&#39;*ndvi+&#39;+str(round(intercept, 2)), color=&#39;black&#39;)<br>    ax.set_title(&#39;Fractional Cover vs NDVI at threshold of &#39;+ str(THRESHOLD))<br><br>    scatter = ax.scatter(x=ndvi, y=fc, edgecolors=&#39;black&#39;)<br><br>    ax.set_xlabel(&#39;Normalized difference vegetation index (NDVI)&#39;)<br>    ax.set_ylabel(&#39;Fractional Cover (fc)&#39;)<br><br>    ax.text(min(ndvi)+0.8*(max(ndvi)-min(ndvi)), min(fc)+0.2*(max(fc)-min(fc)),s=&#39;R^2 = {}&#39;.format(round((r_value**2), 4)), fontdict={&#39;fontsize&#39;:14, &#39;fontweight&#39;:&#39;bold&#39;})<br>    f.savefig(&#39;fc_vs_ndvi_plots/fc_vs_ndvi_&#39;+str(self.plot_counter)+&#39;.jpg&#39;)<br>    f.show()</pre><p>See the <a href="https://github.com/spiyer99/spiyer99.github.io/blob/master/nbs/blog_post_segment_satellite_ndvi.ipynb">full code</a> for more details.</p><p>We can change the threshold value set earlier and see how that affects the regression. The paper notes that we should select the threshold value which has the best regression model.</p><p>We’ll run this code for all images and plot the results.</p><p>From this we can see that the highest R² is associated with a threshold of 0.45.</p><p>We can apply this threshold to all images. Any NDVI value greater than 0.45 is vegetation. Anything below 0.45 is soil.</p><h3>Create Binary Array</h3><p>Using the 0.45 threshold we can create a binary array. In the array 1 denotes vegetation. 0 denotes soil.</p><p>This bit of code does exactly that. The threshold is defined by in the __init__ function. See the <a href="https://github.com/spiyer99/spiyer99.github.io/blob/master/nbs/blog_post_segment_satellite_ndvi.ipynb">full code</a> for details.</p><pre>def create_mask(self, red_file):<br><br>    nir_file = os.path.dirname(red_file) + &#39;/nir.tif&#39;<br>    band_red = rasterio.open(red_file)<br>    band_nir = rasterio.open(nir_file)<br>    red = band_red.read(1).astype(&#39;float64&#39;)<br>    nir = band_nir.read(1).astype(&#39;float64&#39;)<br><br>    # get raw ndvi and save as jpg<br>    self.raw_ndvi = np.where((nir+red)==0., np.nan, (nir-red)/(nir+red))<br>    <br>    # create canopy cover mask<br>    self.canopy_cover = np.where(np.isnan(self.raw_ndvi), np.nan, np.where(self.raw_ndvi&lt;self.ndvi_threshold, 0, 1))<br>    self.canopy_cover = np.ma.masked_where(np.isnan(self.canopy_cover), self.canopy_cover)<br><br>    # show ndvi mask and save it as jpg<br>    print(&#39;canopy cover&#39;)<br>    print(np.unique(self.canopy_cover))<br><br>    return self.canopy_cover</pre><p>Here’s the output in a cleaner form. The first row is the thresholded canopy cover. The second row is the RGB satellite image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tHzwUHSFYxc3oGBs.png" /><figcaption>Image by Author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9M-SwWlQqQQ-JygI.png" /><figcaption>Image by Author</figcaption></figure><p>We can see that it does a pretty decent job of separating canopy cover from soil. This is much better than my <a href="https://towardsdatascience.com/segment-satellite-images-using-rasterio-and-scikit-learn-fc048f465874">previous attempt</a> with K-Means Clustering.</p><h3>Conclusion</h3><p>In this blog post, I described a way you can segment satellite imagery using NDVI and rasterio.</p><p>I did this work for a <a href="https://flurosat.com/">small startup in Sydney</a>. I could not have done this without their help. I learned so much from them.</p><p>The full code is on <a href="https://github.com/spiyer99/spiyer99.github.io/blob/master/nbs/blog_post_segment_satellite_ndvi.ipynb">Github</a>.</p><p>I hope this helps people out there. Please reach out to me on <a href="https://twitter.com/neeliyer11">twitter</a> if I’ve made a mistake. Thanks!</p><p><em>Originally published at </em><a href="https://spiyer99.github.io/Segment-Satellite-Imagery/"><em>https://spiyer99.github.io</em></a><em> on January 31, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6dcae02a044b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/segment-satellite-imagery-using-ndvi-and-rasterio-6dcae02a044b">Segment Satellite Imagery using NDVI and Rasterio</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Segment Satellite Images using Rasterio and Scikit-Learn]]></title>
            <link>https://medium.com/data-science/segment-satellite-images-using-rasterio-and-scikit-learn-fc048f465874?source=rss-e7c5e1c9cae1------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc048f465874</guid>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[rasterio]]></category>
            <category><![CDATA[remote-sensing]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[satellite-imagery]]></category>
            <dc:creator><![CDATA[Neel Iyer]]></dc:creator>
            <pubDate>Sun, 10 Jan 2021 00:00:00 GMT</pubDate>
            <atom:updated>2021-01-18T23:18:15.269Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-on Tutorials</a></h4><h4>Use KMeans clustering to segment satellite imagery by land cover/land use</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8M1VUM8kGPkwBGF0.png" /><figcaption>Image by Author</figcaption></figure><p>Recently, I applied KMeans clustering to Satellite Imagery and was impressed by the results. I’ll tell you the tricks I learned so you don’t waste your time.</p><p>Things to note:</p><ul><li>Use <a href="https://rasterio.readthedocs.io/en/latest/">rasterio</a> not <a href="https://gdal.org/">gdal</a>. Rasterio is more <a href="https://blog.mapbox.com/rasterio-gdal-especially-for-python-programmers-553dabf7dd71">pythonic</a>.</li><li>For this example I’ll be using <a href="http://www.terravion.com/">Terravion imagery</a>. This gives high resolution low level satellite imagery. The Terravion imagery comes in 8 different <a href="https://gsp.humboldt.edu/OLM/Courses/GSP_216_Online/lesson3-1/bands.html">bands</a>.</li><li>I’ll have 3 clusters. These will include: Canopy cover (trees, vegetation, etc. ), Soil and Background.</li></ul><h3>KMeans Explanation</h3><p>I made an infographic to explain KMeans in simple english. Check it out on <a href="https://www.reddit.com/r/learnmachinelearning/comments/kipra3/i_made_an_infographic_to_summarise_kmeans/">reddit</a> or <a href="https://twitter.com/neeliyer11/status/1341663856783810560">twitter</a>.</p><h3>Stack Bands</h3><p>Each Terravion image has the following bands (yours may vary):</p><ul><li>red2.tif</li><li>alpha.tif</li><li>tirs.tif</li><li>blue.tif</li><li>nir.tif</li><li>red.tif</li><li>green.tif</li><li>green2.tif</li></ul><p>You’ll need to stack all the bands before doing anything. Here’s some code for stacking the bands. It was taken from <a href="https://gis.stackexchange.com/questions/223910/using-rasterio-or-gdal-to-stack-multiple-bands-without-using-subprocess-commands">this post</a>.</p><pre>def stack_bands(files):<br>  img_fp = &#39;sentinel_bands.tif&#39;<br>  # Read metadata of first file and assume all other bands are the same<br>  with rasterio.open(files[0]) as src0:<br>      meta = src0.meta<br><br>  # Update metadata to reflect the number of layers<br>  meta.update(count = len(files))<br><br>  # Read each layer and write it to stack<br>  with rasterio.open(img_fp, &#39;w&#39;, **meta) as dst:<br>      for id, layer in enumerate(files, start=1):<br>          with rasterio.open(layer) as srclassifer:<br>              dst.write_band(id, srclassifer.read(1))<br><br>  return img_fp</pre><h3>Fit KMeans</h3><p>Now we’ll need to fit the KMeans classifier to the data. What I found worked best was fitting the KMeans classifier on a few images. Ideally images with a distinct difference between canopy cover and soil.</p><p>The train function will take in a md5 code (eg. 9bbd67cfb0d660551d74decf916b2df2) and a date string (eg. 20190223T130237). It will find the matching image in the dataset and fit the KMeans classifier on that image.</p><p>A few important things:</p><ul><li>I’m not using the alpha and blue channels. They proved to be useless.</li><li>I’m using red2 and green2. red and green didn’t prove to be very useful.</li></ul><p>This training code was adapted from this <a href="https://github.com/wathela/Sentinel2-clustering/blob/master/Sentinel2_Image_clustering.ipynb">github repo</a>.</p><pre>from sklearn.cluster import KMeans<br>from rasterio.plot import reshape_as_image<br>import matplotlib.cm as cm<br>from sklearn import cluster<br><br>def get_date_from_text(text):<br>  text = text.split(&#39;/&#39;)[-2].split(&#39;_&#39;)[4]<br>  datetime_date = datetime.strptime(text, &#39;%Y%m%dT%H%M%S&#39;).date()<br>  return datetime_date<br><br># no alpha and blue imgs<br>def get_files_from_code(code, date):<br>  files = glob.glob(&#39;/content/canopy_cover_cotton_*&#39;+code+&#39;_&#39;+date+&#39;*TERRAV_PLN/*.tif&#39;)<br>  files = sorted(files, key = lambda x: get_date_from_text(x))<br>  files = [i for i in files if os.path.basename(i).split(&#39;.&#39;)[0] in (&#39;red2&#39;, &#39;green2&#39;, &#39;nir&#39;, &#39;tirs&#39;)]<br>  return files<br><br>def train(k = 3, classifer = None, date = &#39;20190223T130237&#39;, code = &#39;9bbd67cfb0d660551d74decf916b2df2&#39;):<br><br>  files = get_files_from_code(code, date)<br><br>  img_fp = stack_bands(files)<br>  img_raster = rasterio.open(img_fp)<br><br>  # Read, enhance and show the image<br>  img_arr = img_raster.read()<br>  vmin, vmax = np.nanpercentile(img_arr, (5,95))  # 5-95% contrast stretch<br><br>  # create an empty array with same dimension and data type<br>  imgxyb = np.empty((img_raster.height, img_raster.width, img_raster.count), img_raster.meta[&#39;dtype&#39;])<br><br>  # loop through the raster&#39;s bands to fill the empty array<br>  for band in range(imgxyb.shape[2]):<br>      imgxyb[:,:,band] = img_raster.read(band+1)<br><br>  # convet to 1d array. 4 cause we have 4 bands here.<br>  img1d = imgxyb[:,:,:4].reshape((imgxyb.shape[0]*imgxyb.shape[1],imgxyb.shape[2]))<br>  # print(img1d.shape)<br><br>  # create an object of the classifier and train it<br>  if(classifer == None):<br>    classifer = cluster.KMeans(n_clusters=k)<br>  # param = cl.fit(img1d[~mask])<br>  param = classifer.fit(img1d)<br><br>  # get the labels of the classes and reshape it x-y-bands shape order (one band only)<br>  img_cl = classifer.labels_<br>  img_cl = img_cl.reshape(imgxyb[:,:,0].shape)<br><br>  # Show the resulting array and save it as jpg image<br>  plt.figure()<br>  plt.imshow(img_cl, cmap=cm.YlOrRd)<br>  plt.axis(&#39;off&#39;)<br>  plt.savefig(&quot;kmeans_train_image.jpg&quot;, bbox_inches=&#39;tight&#39;)<br>  plt.show()<br><br>  return classifer</pre><pre># train model<br>train_dates = [&#39;20190502T113544&#39;, &#39;20190502T113205&#39;]<br>train_codes = [&#39;ea36717ca661ca3cca59d5ea43a81afc&#39;, &#39;9bbd67cfb0d660551d74decf916b2df2&#39;]<br>model = None<br><br>for i in range(min(len(train_codes), len(train_dates))):<br>  model = train(classifer = model, date = train_dates[i], code = train_codes[i])</pre><p>Here’s the training output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/221/0*Kef07ROIhEs_09Ek.png" /><figcaption>Image by Author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/239/0*ChTscmI7MJLXmD-J.png" /><figcaption>Image by Author</figcaption></figure><p>It looks pretty good!</p><h3>Prediction</h3><p>Now we can run predictions on our model and see how it does.</p><p>This function will take in the stacked bands and saved model. it will then run the KMeans model on a new image. The prediction output will be saved as a jpg file.</p><p>This saved jpg is for visualisation purposes only. Don’t use it for further calculations. I accidentally did that and got very confused.</p><pre>from tqdm.notebook import tqdm<br><br>def get_date_from_orig_image(text):<br>  text = os.path.basename(text).split(&#39;.&#39;)[0].split(&#39;_&#39;)[-1]<br>  datetime_date = datetime.strptime(text, &#39;%Y%m%dT%H%M%S&#39;).date()<br>  return datetime_date<br><br>def get_orig_img_list(code, date):<br>  original_images = glob.glob(&#39;/content/drive/My Drive/flurosat/*&#39;+code+&#39;_&#39;+date+&#39;*.*g&#39;)<br>  original_images = sorted(original_images, key = lambda x: get_date_from_orig_image(x))<br>  return original_images[0]<br><br># predict using new k means method<br>def predict(model, img_fp):<br><br>  elmanagel = rasterio.open(img_fp)<br><br>  # Read, enhance and show the image<br>  elman_arr = elmanagel.read()<br>  vmin, vmax = np.nanpercentile(elman_arr, (5,95))  # 5-95% contrast stretch<br><br>  # create an empty array with same dimensions and data type <br>  elman_xyb = np.empty((elmanagel.height, elmanagel.width,elmanagel.count), elmanagel.meta[&#39;dtype&#39;])<br><br>  # loop through the raster bands and fill the empty array in x-y-bands order<br>  for band in range(elman_xyb.shape[2]):<br>      elman_xyb[:,:,band] = elmanagel.read(band+1)<br><br>  # convert to 1d array<br>  elman_1d = elman_xyb[:,:,:elman_xyb.shape[2]].reshape(elman_xyb.shape[0]*elman_xyb.shape[1], elman_xyb.shape[2])<br><br>  # predict the clusters in the image <br>  pred = model.predict(elman_1d)<br><br>  # reshape the 1d array predictions to x-y-bands shape order (only one band)<br>  elman_cul = pred<br>  elman_cul = elman_cul.reshape(elman_xyb[:,:,0].shape)<br><br>  return elman_cul<br><br>def predictions_driver(code, date, iterations):<br><br>  files = get_files_from_code(code, date)<br>  original_image = get_orig_img_list(code, date)<br><br>  # # rasterio stack all bands<br>  img_fp = stack_bands(files)<br><br>  # predict on trained model<br>  kmeans_predictions = predict(model, img_fp)<br><br>  # save kmeans<br>  plt.imsave(&#39;kmeans_output/&#39;+str(code)+&#39;_&#39;+str(iterations)+&#39;_k_means.jpg&#39;, kmeans_predictions, cmap=cm.YlOrRd)<br><br>  # save original image<br>  img = PIL.Image.open(original_image)<br>  img.save(&#39;kmeans_output/&#39;+str(code)+&#39;_&#39;+str(iterations)+&#39;_original_image.jpg&#39;)<br><br>  return<br><br>for i,combination in enumerate(tqdm(sorted_combinations)):<br>  date = combination.split(&#39;_&#39;)[-1]<br>  code = combination.split(&#39;_&#39;)[0]<br>  predictions_driver(code, date, i)</pre><h3>Create Grid</h3><p>Now we can create an image grid from the saved images jpgs we obtained previously.</p><p>This makes it far easier to see if KMeans was actually correct.</p><pre>codes = list(set([i.split(&#39;_&#39;)[0] for i in combinations]))<br><br>for code in codes:<br><br>  k_means = glob.glob(&#39;kmeans_output/*&#39;+str(code)+&#39;*k_means.*g&#39;)<br>  k_means = sorted(k_means, key = lambda x: int(os.path.basename(x).split(&#39;.&#39;)[0].split(&#39;_&#39;)[-3]))<br>  k_means = [PIL.Image.open(i) for i in k_means]<br><br>  original_imgs = glob.glob(&#39;kmeans_output/*&#39;+str(code)+&#39;*original_image.*g&#39;)<br>  original_imgs = sorted(original_imgs, key = lambda x: int(os.path.basename(x).split(&#39;.&#39;)[0].split(&#39;_&#39;)[-3]))<br>  original_imgs = [PIL.Image.open(i) for i in original_imgs]<br>  <br>  full_list = k_means + original_imgs<br>  imshow_func(createImageGrid(full_list, rows = 2, scale = 0.1))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9_E6DHJPzkECGyK9.png" /><figcaption>Image by Author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SU7ASySVpWnoi739.png" /><figcaption>Image by Author</figcaption></figure><p>The KMeans output is on the 1st row. The RBG original image is on the 2nd row. We can see that it does a decent job at separating canopy cover from soil.</p><p>More satellite imagery would be required to comprehensively assess its performance. But this is a decent start.</p><h3>Conclusion</h3><p>I did this work for a <a href="https://flurosat.com/">small startup in Sydney</a>. I learned so much from the experienced professionals at this startup. I couldn’t have created this without their help.</p><p>I hope this post helps someone out there. It certaintly would’ve helped me when I started. The full code can be found on <a href="https://github.com/spiyer99/spiyer99.github.io/blob/master/nbs/canopy_cover_blog_post.ipynb">Github</a></p><p>If I’ve made a mistake please reach out to me on <a href="https://twitter.com/neeliyer11">twitter</a></p><p><em>Originally published at </em><a href="https://spiyer99.github.io/Kmeans-Clustering-Satellite-Imagery/"><em>https://spiyer99.github.io</em></a><em> on January 10, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc048f465874" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/segment-satellite-images-using-rasterio-and-scikit-learn-fc048f465874">Segment Satellite Images using Rasterio and Scikit-Learn</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Transformation in PySpark]]></title>
            <link>https://medium.com/data-science/data-transformation-in-pyspark-6a88a6193d92?source=rss-e7c5e1c9cae1------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a88a6193d92</guid>
            <category><![CDATA[data-transformation]]></category>
            <category><![CDATA[introduction-to-pyspark]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Neel Iyer]]></dc:creator>
            <pubDate>Sun, 06 Sep 2020 00:00:00 GMT</pubDate>
            <atom:updated>2021-04-06T21:28:32.969Z</atom:updated>
            <content:encoded><![CDATA[<h4>A walkthrough of Data Transformations in PySpark</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uJ0hRRJMtUYV1h-FV3L2Dw.jpeg" /><figcaption>Image by <a href="https://www.pexels.com/@markusspiske?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Markus Spiske</a> from <a href="https://www.pexels.com/photo/laptop-office-internet-technology-177598/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><p>Data is now growing faster than processing speeds. One of the many solutions to this problem is to parallelise our computing on large clusters. Enter PySpark.</p><p>However, PySpark requires you to think about data differently.</p><p>Instead of looking at a dataset row-wise. PySpark encourages you to look at it column-wise. This was a difficult transition for me at first. I’ll tell you the main tricks I learned so you don’t have to waste your time searching for the answers.</p><h3>Dataset</h3><p>I’ll be using the <a href="https://www.kaggle.com/epa/hazardous-air-pollutants">Hazardous Air Pollutants</a> dataset from Kaggle.</p><p>This Dataset is 8,097,069 rows.</p><pre>df = spark.read.csv(‘epa_hap_daily_summary.csv’,inferSchema=True, header =True)<br>df.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N1H5dyVCVGmnDsCi.png" /><figcaption>Image by Author</figcaption></figure><h3>Conditional If Statement</h3><p>The first transformation we’ll do is a conditional if statement transformation. This is as follows: if a cell in our dataset contains a particular string we want to change the cell in another column.</p><p>Basically we want to go from this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/0*1Lk9C5VlONNqLBKQ.png" /><figcaption>Image by Author</figcaption></figure><p>To this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*iA8W-xgwlaWjIi_y.png" /><figcaption>Image by Author</figcaption></figure><p>If local site name contains the word police then we set the is_police column to 1. Otherwise we set it to 0.</p><p>This kind of condition if statement is fairly easy to do in Pandas. We would use <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.where.html">pd.np.where</a> or <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.apply.html">df.appl</a>y. In the worst case scenario, we could even iterate through the rows. We can’t do any of that in Pyspark.</p><p>In Pyspark we can use the <a href="https://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html#pyspark.sql.functions.when">F.when</a> statement or a <a href="https://spark.apache.org/docs/2.2.0/api/python/pyspark.sql.html#pyspark.sql.functions.udf">UDF</a>. This allows us to achieve the same result as above.</p><pre>from pyspark.sql import functions as F</pre><pre>df = df.withColumn(&#39;is_police&#39;,\<br>     F.when(\<br>     F.lower(\<br>     F.col(&#39;local_site_name&#39;)).contains(&#39;police&#39;),\<br>     F.lit(1)).\<br>                    otherwise(F.lit(0)))</pre><pre>df.select(&#39;is_police&#39;, &#39;local_site_name&#39;).show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*qjs7UJOPHH3HjuNk.png" /><figcaption>Image by Author</figcaption></figure><p>Now suppose we want to extend what we’ve done above. This time if a cell contains any one of 3 strings then we change the corresponding cell in another column.</p><p>If any one of strings: &#39;Police&#39;, &#39;Fort&#39; , &#39;Lab&#39; are in the local_site_name column then we&#39;ll mark the corresponding cell as High Rating.</p><p>The<a href="https://spark.apache.org/docs/2.2.0/api/python/pyspark.sql.html#pyspark.sql.Column.like">rlike</a> function combined with the F.when function we saw earlier allows to us to do just that.</p><pre>parameter_list = [&#39;Police&#39;, &#39;Fort&#39; , &#39;Lab&#39;]</pre><pre>df = df.withColumn(&#39;rating&#39;,\<br>     F.when(\<br>     F.col(&#39;local_site_name&#39;).rlike(&#39;|&#39;.join(parameter_list)),\<br>     F.lit(&#39;High Rating&#39;)).\<br>     otherwise(F.lit(&#39;Low Rating&#39;)))</pre><pre>df.select(&#39;rating&#39;, &#39;local_site_name&#39;).show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/0*7dOZco12eTm-vqWR.png" /><figcaption>Image by Author</figcaption></figure><p>F.when is actually useful for a lot of different things. In fact you can even do a chained F.when:</p><pre>df = df.withColumn(&#39;rating&#39;, F.when(F.lower(F.col(&#39;local_site_name&#39;)).contains(&#39;police&#39;), F.lit(&#39;High Rating&#39;))\<br>                              .when(F.lower(F.col(&#39;local_site_name&#39;)).contains(&#39;fort&#39;), F.lit(&#39;High Rating&#39;))\<br>                              .when(F.lower(F.col(&#39;local_site_name&#39;)).contains(&#39;lab&#39;), F.lit(&#39;High Rating&#39;))\<br>                              .otherwise(F.lit(&#39;Low Rating&#39;)))</pre><pre>df.select(&#39;rating&#39;, &#39;local_site_name&#39;).show(</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/0*5Ho3BixJHbII5EEo.png" /><figcaption>Image by Author</figcaption></figure><p>This achieves exactly the same thing we saw in the previous example. However, it’s more code to write and it’s more code to maintain.</p><p>I prefer the rlike method discussed above.</p><h3>Remove whitespace</h3><p>Whitespace can be really annoying. It really affects string matches and can cause unnecessary bugs in queries.</p><p>In my opinion it’s a good idea to remove whitespace as soon as possible.</p><p><a href="https://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html#pyspark.sql.functions.trim">F.trim</a>allows us to do just that. It will remove all the whitespace for every row in the specified column.</p><pre>df = df.withColumn(&#39;address&#39;, F.trim(F.col(&#39;address&#39;)))<br>df.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qYMCI8PfqN-YwXTu.png" /><figcaption>Image by Author</figcaption></figure><h3>Remove Null Rows for a Particular Column</h3><p>Suppose we want to remove null rows on only one column. If we encounter NaN values in the pollutant_standard column drop that entire row.</p><p>This can accomplished fairly simply.</p><pre>filtered_data = df.filter((F.col(&#39;pollutant_standard&#39;).isNotNull())) # filter out nulls<br>filtered_data.count()</pre><p>The conditional OR parameter allows to remove rows where we event_type or site_num are NaN.</p><p>This is <a href="https://stackoverflow.com/questions/3154132/what-is-the-difference-between-logical-and-conditional-and-or-in-c">referred</a> to as `|`.</p><pre>filtered_data = df.filter((F.col(&#39;event_type&#39;).isNotNull()) | (F.col(&#39;site_num&#39;).isNotNull())) # filter out nulls<br>filtered_data.count()</pre><p><a href="https://spark.apache.org/docs/2.2.0/api/python/pyspark.sql.html#pyspark.sql.DataFrame.dropna">df.na.drop</a> allows us to remove rows where all our columns are NaN.</p><pre>filtered_data = df.na.drop(how = &#39;all&#39;) # filter out nulls<br>filtered_data.show()</pre><h3>Conclusion</h3><p>PySpark is still fairly a new language. Probably as a result of that there isn’t a lot of help on the internet. With something like Pandas or R there’s a wealth of information out there. With Spark that’s not the case at all.</p><p>So I hope these bits of code help someone out there. They certainly would’ve helped me and saved me a lot of time. The transformations I went through might seem small or trivial but there aren’t a lot of people talking about this stuff when it pertains to Spark. I hope this helps you in some way.</p><p>If I’ve made a mistake or you’d like reach out to me feel free to contact me on <a href="https://twitter.com/neeliyer11">twitter</a>.</p><p><em>Originally published at </em><a href="https://spiyer99.github.io/Pyspark-Hacks/"><em>https://spiyer99.github.io</em></a><em> on September 6, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a88a6193d92" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/data-transformation-in-pyspark-6a88a6193d92">Data Transformation in PySpark</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Instance Segmentation Web App]]></title>
            <link>https://medium.com/data-science/instance-segmentation-web-app-63016b8ed4ae?source=rss-e7c5e1c9cae1------2</link>
            <guid isPermaLink="false">https://medium.com/p/63016b8ed4ae</guid>
            <category><![CDATA[detectron2]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[instance-segmentation]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Neel Iyer]]></dc:creator>
            <pubDate>Sun, 19 Jul 2020 00:00:00 GMT</pubDate>
            <atom:updated>2021-03-05T04:48:23.275Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building a Web App for Instance Segmentation using Docker, Flask and Detectron2</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FVVLkB-vYmCc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DVVLkB-vYmCc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FVVLkB-vYmCc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b9a38351c40e5143f548e11d4c10db3b/href">https://medium.com/media/b9a38351c40e5143f548e11d4c10db3b/href</a></iframe><p><a href="https://github.com/facebookresearch/detectron2">Detectron2</a> offers state of the art instance segmentation models. It’s very <a href="https://detectron2.readthedocs.io/notes/benchmarks.html">quick to train</a> and offers very good results.</p><p>Model training is fairly straightforward. There are many <a href="https://github.com/facebookresearch/detectron2/blob/master/GETTING_STARTED.md">tutorials</a> to help you there. Deploying the model to a web app is a different story. When I was trying to do this I didn’t find a lot of help on the internet.</p><p>So, in this post we’ll create a web app for detectron2’s instance segmentation.</p><h3>Backend</h3><p>First, we’ll create the machine learning backend. This will use basic <a href="https://flask.palletsprojects.com/en/1.1.x/">flask</a>. We’ll start from some fairly standard <a href="https://github.com/realpython/flask-boilerplate/blob/master/app.py">boilerplate code</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e629d9b9d7e3651c104eb70ade0f6c2/href">https://medium.com/media/9e629d9b9d7e3651c104eb70ade0f6c2/href</a></iframe><p>This app will simply render the template index.html. I&#39;ve specified the port manually.</p><p>Next we’ll add functions to get the image. We want to be able to upload an image to the website. We also want to be able to supply the website with a url and the image will be downloaded automatically. I’ve created the code do to exactly that below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e4fd06c09bcb54960b66e9ee9a460bb/href">https://medium.com/media/9e4fd06c09bcb54960b66e9ee9a460bb/href</a></iframe><p>This piece of code allows us to upload an image into the backend (POST request). Or we can supply the backend with a url and it will download the image automatically (GET request). The code also converts the image to a jpg. I couldn&#39;t do inference on a png image using detectron2. So we&#39;ll have to convert to a jpg.</p><p>If the code can’t download the image for whatever reason — It will return the failure.html template. This will basically just be a simple html page saying there was an error in retrieving the image.</p><p>Also, I’ve specified a different @app.route. This will need to reflected in the index.html file.</p><h3>Frontend</h3><p>Now I’ll create the frontend html code. This inferface alows the user to upload an image or specify a url to an image.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/de742f5101c0bb16617cd14c89796edf/href">https://medium.com/media/de742f5101c0bb16617cd14c89796edf/href</a></iframe><p>There’s not much to it. We create a simple form and tell it to link to the @app.route(‘/detect’) flask code. We also need to specify the method. If the user is uploading an image, it’s POST. If the user is giving us the url to an image, it’s GET.</p><p>The failure.html template is even simpler.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/14c8473b06c862acaf65ecbd6c89facf/href">https://medium.com/media/14c8473b06c862acaf65ecbd6c89facf/href</a></iframe><p>Now we can move on to the actual deep learning part.</p><p>In this part, we’ll get a detectron2 pretrained model to do inference on an image. Then we’ll link it to our existing backend.</p><p>This part is slightly more involved. We’ll create a new class called Detector. In that we&#39;ll create the cfg that is required for detectron2. Then we&#39;ll create another function that will do the inference on an image.</p><p>I’ll be using the mask rcnn pretrained model trained on the <a href="https://cocodataset.org/">COCO</a> dataset. It will use a <a href="https://towardsdatascience.com/an-overview-of-resnet-and-its-variants-5281e2f56035">ResNet</a>+ <a href="https://towardsdatascience.com/review-fpn-feature-pyramid-network-object-detection-262fc7482610">FPN</a> backbone. This model is said to obtain the <a href="https://github.com/facebookresearch/detectron2/blob/master/MODEL_ZOO.md#common-settings-for-coco-models">best speed/accuracy tradeoff</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/24087c4b528d9aef0664825bdc461705/href">https://medium.com/media/24087c4b528d9aef0664825bdc461705/href</a></iframe><p>This piece of code pretty much does everything we need it to do for inference. We just need to specify the path to the pretrained model that we downloaded.</p><p>It gets the config that corresponds our pretrained model automatically from the model zoo. It should also get the pretrained model from the model zoo as well. But I found that this doesn’t really work in docker — at least for me.</p><p>The next step is to integrate this Detector class into our existing script.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31443d0635548d2fcb4bf39e7027df67/href">https://medium.com/media/31443d0635548d2fcb4bf39e7027df67/href</a></iframe><p>In this piece of code I’ve added in the Detector class we created previously. I&#39;ve also created a function called run_inference. This is where the backend will run the detectron2 model on an image. It will take in an image path and call the detectron2 model through the Detector class we created earlier.</p><p>The run_inference function will return an image once instance segmentation is completed. I had to do some hacky stuff with the result fo the run_inference function to get it to work. result_img is pasted into a file object and that file object is returned as a png. There might be a better way to do this though.</p><p>The final step is creating a docker container for our code. Then we’ll deploy the docker container locally.</p><p>Thankfully, detectron2 has already created a <a href="https://github.com/facebookresearch/detectron2/blob/master/docker/Dockerfile">dockerfile</a> for us.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4ec64e35bfe45650857ae268c26ab86b/href">https://medium.com/media/4ec64e35bfe45650857ae268c26ab86b/href</a></iframe><p>I’ve bascially used the dockerfile supplied in detectron2’s <a href="https://github.com/facebookresearch/detectron2/blob/master/docker/Dockerfile">github repo</a>. But I made a few changes.</p><p>I’ve added a <a href="https://github.com/spiyer99/detectron2_web_app/blob/master/requirements.txt">requirements.txt</a> file. I do a pip install from that requirements file. That installs a few libraries that we need for this to work. I&#39;ve also changed the command to start the app.py script we created earlier. That will start the flask application and render the index.html template.</p><p>Now we can start the docker container. We can do this using the following:</p><pre>docker build . -f Dockerfile -t detectron2 &amp;&amp;\<br>docker run -d -p 8080:8080 detectron2</pre><p>That will build the current dockerfile in the current working directory. Then it will run that image on port 8080. That’s the port that I specified earlier.</p><p>But the issue with that if you keep running those commands over time you’ll have too many docker images and containers on your computer. This can take up a <a href="https://jimhoskins.com/2013/07/27/remove-untagged-docker-images.html">lot of space</a> on your computer.</p><p><a href="https://jimhoskins.com/2013/07/27/remove-untagged-docker-images.html">Jim Hoskins</a> has a pretty elegant solution to this, which I adapted for my purposes.</p><p>I created a nice shell script which:</p><ul><li>stops all the containers</li><li>removes untagged containers</li><li>builds a new container</li><li>runs the container on port 8080</li><li>shows the trailing logs for the container</li></ul><p>This script is <em>incredibly</em> useful.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c775098cba98016c18ebce0951c1e419/href">https://medium.com/media/c775098cba98016c18ebce0951c1e419/href</a></iframe><p>Run this script from terminal. It will build and run the detectron2 web app. The web app should show up at localhost:8080 on your browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oPyJY-nhOHRcvPxV.png" /><figcaption>Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*kB2OQ5fAspcIbB2-JOISzQ.jpeg" /><figcaption>Source image by <a href="https://unsplash.com/@nabeelsyed">Nabeel Syed</a> on <a href="https://unsplash.com/photos/Jk3-Uhdwjcs">Unsplash</a></figcaption></figure><p>It works!</p><p>The code for this can be found on <a href="https://github.com/spiyer99/detectron2_web_app">github</a>.</p><p><em>Originally published at </em><a href="https://spiyer99.github.io/Detectron2-Web-App/"><em>https://spiyer99.github.io</em></a><em> on July 19, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63016b8ed4ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/instance-segmentation-web-app-63016b8ed4ae">Instance Segmentation Web App</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Star Wars: Episode IV (1977) Remastered]]></title>
            <link>https://medium.com/data-science/remastering-star-wars-using-deep-learning-b21713928a53?source=rss-e7c5e1c9cae1------2</link>
            <guid isPermaLink="false">https://medium.com/p/b21713928a53</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[star-wars]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Neel Iyer]]></dc:creator>
            <pubDate>Fri, 19 Jun 2020 00:00:00 GMT</pubDate>
            <atom:updated>2020-09-19T10:36:55.350Z</atom:updated>
            <content:encoded><![CDATA[<h3>Star Wars Episode IV (1977) Remastered</h3><h4>A Deep Learning pipeline to remaster the deleted scenes</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nI8Sd-4HVh5aHOXZa-wpeQ.jpeg" /><figcaption>Image by <a href="https://unsplash.com/@koalamoose">Agnieszka Kowalczyk</a> on <a href="https://unsplash.com/photos/c0VRNWVEjOA">Unsplash</a></figcaption></figure><p><strong>TLDR</strong></p><p>Here’s the output from the model. Left is remastered output. Right is original video.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FH7s9BUKVL0M%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DH7s9BUKVL0M&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FH7s9BUKVL0M%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d084c9763f50a532a0914c536355198d/href">https://medium.com/media/d084c9763f50a532a0914c536355198d/href</a></iframe><p>Full Jupyter Notebook for training and running inference is is available on <a href="https://github.com/spiyer99/spiyer99.github.io/blob/master/nbs/star_wars_remastery.ipynb">Github</a></p><p><strong>A New Hope for the deleted scenes</strong></p><p>I’m a huge Star Wars fan. And like a lot of Star Wars fans I’ve been getting into <a href="https://www.imdb.com/title/tt0458290/">Star Wars: The Clone Wars</a> on Cartoon Network and Disney+. It’s a phenomenal show.</p><p>But I’m always annoyed by the drop in video quality when I watch the older stuff. For example, here are the deleted scenes from Star Wars: Episode IV: A New Hope (1977). This was the very first Star Wars to be created.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Ff00IkrWvur4%3Fstart%3D162%26feature%3Doembed%26start%3D162&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Df00IkrWvur4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Ff00IkrWvur4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/806c6b05f3d9feaff873ccd3cfaf72a5/href">https://medium.com/media/806c6b05f3d9feaff873ccd3cfaf72a5/href</a></iframe><p>There’s these weird black specs that keep popping up. Small wonder why these are the <em>deleted</em> scenes.</p><p>Apparently those weird specs are called <a href="https://en.wikipedia.org/wiki/Cue_mark">cue marks</a>. They’re marks that come from scratches on film. Star Wars is a fantastic series, but it’s also fantastically <em>old</em>.</p><p>Deep Learning has recently been used for video restoration. The results have been very promising. <a href="https://github.com/jantic/DeOldify">Deoldify</a> for example, allows users to colorize old videos and images. <a href="https://www.youtube.com/watch?v=P0fMwA3X5KI">NVIDIA’s Noise2Noise model</a> allows people to restore old images to their former glory.</p><p>But so far there’s nothing I know of that can specifically remove ‘cue marks’ and grainy spots from old film. So let’s build it!</p><h3>Creating the Dataset</h3><p>Creating the dataset was tricky — but still doable. Here’s what I did. I downloaded high quality videos into from youtube. Then I ruined them. I added black specs and reduced the resolution of the video. <a href="https://ffmpeg.org/">Ffmpeg</a> was very useful in doing this.</p><p>First we’ll download the video.</p><pre>youtube-dl --format best -o seinfeld.mp4 <a href="https://www.youtube.com/watch?v=nEAO60ON7yo">https://www.youtube.com/watch?v=nEAO60ON7yo</a></pre><p>I’m using this video. I’m using a clip from <a href="https://en.wikipedia.org/wiki/Seinfeld">Seinfeld</a>. Cause why not?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FnEAO60ON7yo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DnEAO60ON7yo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FnEAO60ON7yo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/03e68d4b54b3f910080d765c06eb57d7/href">https://medium.com/media/03e68d4b54b3f910080d765c06eb57d7/href</a></iframe><p>Then we’ll need to ruin it. To do this I downloaded a grainy film overlay from youtube. Then I overlayed the video using ffmpeg with the blend setting set to softlight. Finding the right blend setting took a lot of trial and error. The ffmpeg <a href="https://ffmpeg.org/documentation.html">docs</a> don’t have a lot of examples.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2903f863ec3d7d1713320872fe0a0c2a/href">https://medium.com/media/2903f863ec3d7d1713320872fe0a0c2a/href</a></iframe><p>Now we have two videos. One in perfect quality and another in crappy quality.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FMaIDJO5ar1c%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DMaIDJO5ar1c&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FMaIDJO5ar1c%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a81cf358b6d236405c7916d81e0f19d1/href">https://medium.com/media/a81cf358b6d236405c7916d81e0f19d1/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fl8Z3T9w0yBY%3Fstart%3D2%26feature%3Doembed%26start%3D2&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dl8Z3T9w0yBY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fl8Z3T9w0yBY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5b9fe404eab39e779be9bc9cfe8b8a31/href">https://medium.com/media/5b9fe404eab39e779be9bc9cfe8b8a31/href</a></iframe><p>Now we’ll extract frames from each video. Initially, I adopted a naive approach for doing this. Where I would do through the video in python and scrape each frame individually. But that took too long. I eventually realised we can use multi-processing here to really speed things up. This was adapted from <a href="https://gist.github.com/HaydenFaulkner/54318fd3e9b9bdb66c5440c44e4e08b8#file-video_to_frames-py">Hayden Faulker’s</a> script.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b568cc93724400c636f4c4816af2ac9f/href">https://medium.com/media/b568cc93724400c636f4c4816af2ac9f/href</a></iframe><p>Great. Now we have two datasets. One of crappy quality images (taken from the ruined video) and one of good quality images (taken from the high quality video). To make the crappy images crappier, I’ll downscale them (this isn’t a necessary step though).</p><pre>def resize_one(img, size):</pre><pre>   targ_sz = resize_to(img, size, use_min = True)<br>   img = img.resize(targ_sz, resample =  PIL.Image.BILINEAR).convert(&#39;RGB&#39;)<br>   return img</pre><p>This is what the crappy and normal images looked like now. Side note: this is a great scene from seinfeld.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/0*fYq2ApvCdd65ItPd.png" /><figcaption>Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/0*KIwkIykgYKvMgvhZ.png" /><figcaption>Image by author</figcaption></figure><p>A quick check shows that we have a dataset of about 10014 files. Pretty good.</p><h3>Neural Network</h3><p>Let’s make the most of those 10014 files by using transforms.</p><p>I added horizontal and vertical flips, zoom changes, lighting changes and rotation changes. With Fastai this is really easy to do.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/079afd3983239706056672dcc46fa442/href">https://medium.com/media/079afd3983239706056672dcc46fa442/href</a></iframe><p>Here are some of the image transforms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/0*OFvtOyHG0r8VDuYT.png" /><figcaption>Image by author</figcaption></figure><p>Not bad!</p><p>We’ll use the <a href="https://www.fast.ai/2019/05/03/decrappify/">NoGAN network</a> pioneered by fastai and jason antic on this data. This code was inspired by <a href="https://github.com/fastai/course-v3/blob/master/nbs/dl1/lesson7-superres.ipynb">lesson 7</a> of the fastai course.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2afba31cfebac315204398c70260c1f3/href">https://medium.com/media/2afba31cfebac315204398c70260c1f3/href</a></iframe><p>I trained the model on <a href="https://colab.research.google.com/">google colab’s free gpus</a>. They’re a great resource and I can’t believe they are free.</p><h3>Training</h3><p>The interesting thing that fastai <a href="https://www.youtube.com/watch?v=9spwoDYwW_I">recommends</a> is increasing the size of your images gradually.</p><p>So at first, you train on small size images, then you upscale your images and retrain on the larger images. It saves you a lot of time. Pretty smart.</p><p>First, we’ll train on images of size 128x128. Because the images are so small I can up the batch size to 64.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e015202a9691660d254b66a8bfc66d33/href">https://medium.com/media/e015202a9691660d254b66a8bfc66d33/href</a></iframe><p>I picked a learning rate of 1e-2 for this. I wanted something aggressive, but still on the safe side of explosion. This has been <a href="https://sgugger.github.io/how-do-you-find-a-good-learning-rate.html">shown</a> to be very useful.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ea7eb2790e0ed06df50365f78fd59ee6/href">https://medium.com/media/ea7eb2790e0ed06df50365f78fd59ee6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PBtoBj9cgm_-cRSM.png" /><figcaption>Image by author</figcaption></figure><p>The network will print the results during training. The input is on the left, the prediction in the middle and the target on the right. The results look very promising!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wbEQJ3I8cyp9TYM9.png" /><figcaption>Image by author</figcaption></figure><p>I resized and trained again. And again. Every time I make the resize slightly larger than it was previously. I moved from 128x128 to 480x480 to the original size of the video frames.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/253e41671ac89fdf24353a8ee2b9c08f/href">https://medium.com/media/253e41671ac89fdf24353a8ee2b9c08f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gJTRIkkHuiPmw6Oe.png" /><figcaption>Image by author</figcaption></figure><p>This was the final train. For this I used pct_start = 0.3. I wanted to learning rate to reduce 70% of the time during training. I prefer a lower learning rate when fine tuning models. The results from this piece of training look really good.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aDytged6tGX0kUDn.png" /><figcaption>Image by author</figcaption></figure><h3>Inference: Applying to Star Wars</h3><p>Once this network had trained, I ran inference. This was more involved than I originally thought.</p><p>I had to download the Star Wars deleted scenes (using <a href="https://github.com/ytdl-org/youtube-dl">youtube-dl</a>) and then extract all the frames in this video. I extracted the frames using the same method previously.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/0*3fazAhXbwYPeVWGq.png" /><figcaption>Image by author</figcaption></figure><p>Then I had to run inference from the learner on each individual frame of the video. That takes a long time.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd9afd7f8b071f2b0b9e9438985b9cc2/href">https://medium.com/media/dd9afd7f8b071f2b0b9e9438985b9cc2/href</a></iframe><p>I added some hacks here.</p><p>First, I added a render factor. This was taken from <a href="https://github.com/jantic/DeOldify">Deoldify</a>. The idea is that I downscale the image and convert it to a square. Then I run inference on that image. The model is more receptive to images that are square shaped. This has been <a href="https://github.com/jantic/DeOldify#stuff-that-should-probably-be-in-a-paper">shown</a> to reduce ‘glitches’ considerably.</p><p>After running inference on the square shaped image I convert it back to its original shape. I found this to reduce glitches and generally result in a smoother video output. I set the render_factor to 40, although it can be higher if we want higher res output. I may need a larger RAM for that though.</p><p>Second, I adjust brightness. This isn’t really a hack. Seems like more of a mistake that I’m correctly manually. For some reason, the model inference results in images that are very low in brightness.</p><p>I suspect it’s something to with the softlight filter we used for ffmpeg earlier. But I&#39;m having to manually correct that here. I&#39;ll need to look into this further.</p><p>Third, I’m using matplotlib’s save functionality. I found fastai’s <a href="https://docs.fast.ai/vision.image.html#Image.save">save image</a> functionality to give me very weird results (Luke’s clothes were fluroscent blue and red). But strangely, matplotlib’s save functionality gives me okay results. I’ll need to look into this. I suspect that I may be losing quality on the image because I’m using matplotlib’s savefig functionality.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f678fdbb90310ec6fafb9d77c0f116e/href">https://medium.com/media/8f678fdbb90310ec6fafb9d77c0f116e/href</a></iframe><p>Here’s some of the outputs from the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/0*GMI6YGn7qJBTMTHA.png" /><figcaption>Images by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/0*gZYY4VnxUM-StEgB.png" /><figcaption>Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/0*ciOw5qZZL8G8B0lG.png" /><figcaption>Image by author</figcaption></figure><p>Then I had to stitch all these frames together to create a video. To do that I initially used ffmpeg but I ended up overloading my RAM. Instead, I used opencv2’s VideoWriter.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3d462508fa276c991f8ccab252b05eb2/href">https://medium.com/media/3d462508fa276c991f8ccab252b05eb2/href</a></iframe><p>Here is the final output.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F2BcoLsuJMP0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D2BcoLsuJMP0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F2BcoLsuJMP0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/592007226507b5353b2b93bc9f6a25dd/href">https://medium.com/media/592007226507b5353b2b93bc9f6a25dd/href</a></iframe><p>And the original video</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Ff00IkrWvur4%3Fstart%3D162%26feature%3Doembed%26start%3D162&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Df00IkrWvur4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Ff00IkrWvur4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/806c6b05f3d9feaff873ccd3cfaf72a5/href">https://medium.com/media/806c6b05f3d9feaff873ccd3cfaf72a5/href</a></iframe><h3>Improvements</h3><ol><li>The sky needs a bit more work. But I like the vibrancy of the background. That is an interesting (and completely unplanned) effect. The goal was to remove the ‘cue marks’ (annoying black specs) from the video. I think it’s done okay in that respect — but there’s still more to do.<br>I like how the network has intensified the sun though. It completely changes the the scene between Luke and Biggs when Biggs says he’s joining the rebellion.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yTpeIJWou2E_1_qh.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*q5Kxj3ljFU8t0wB9.png" /><figcaption>Original Frame (left). Output from network (right)</figcaption></figure><p>2. There’s a weird horizontal bar line that shows up around the 22 second mark. I didn&#39;t add any horizontal bars in the training set so it&#39;s completely understandable that the network didn&#39;t remove that at all. But in the future, I&#39;ll need to add more horizontal bars to my training set to fix these.</p><p>3. I’m also thinking of doing more super-resolution on the video. It would be nice to show a young Luke Skywalker in high quality. To do that I could resize the images before training further. I’ve already downscaled the image, but potentially I could downscale it further.<br>Alternatively, to achieve superres I could potentially use a ready-made upscaler such as <a href="https://github.com/AlphaAtlas/VapourSynth-Super-Resolution-Helper">VapourSynth</a>. This is probably the best option as the original video is already in poor quality.</p><p>4. Inference is also an issue. It tends to overload memory and crash. The result is that 42 seconds is the longest I could get for this video. I&#39;m not completely sure how to solve this problem. But I&#39;ll need to solve it if I&#39;m going to be using this further.</p><p>So much to do!</p><p>Full code available on <a href="https://github.com/spiyer99/spiyer99.github.io/blob/master/nbs/star_wars_remastery.ipynb">Github</a></p><p><em>Originally published at </em><a href="https://spiyer99.github.io/Star-Wars-Remastery/"><em>https://spiyer99.github.io</em></a><em> on June 19, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b21713928a53" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/remastering-star-wars-using-deep-learning-b21713928a53">Star Wars: Episode IV (1977) Remastered</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>