<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ahmad Sachal on Medium]]></title>
        <description><![CDATA[Stories by Ahmad Sachal on Medium]]></description>
        <link>https://medium.com/@ahmadsachal7?source=rss-85ddcd6488de------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bLohUvcj9SprdlufhnvyBg.png</url>
            <title>Stories by Ahmad Sachal on Medium</title>
            <link>https://medium.com/@ahmadsachal7?source=rss-85ddcd6488de------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 21:32:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ahmadsachal7/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why Did I Choose WebDataset for Training on 50TB of Data?]]></title>
            <link>https://medium.com/red-buffer/why-did-i-choose-webdataset-for-training-on-50tb-of-data-98a563a916bf?source=rss-85ddcd6488de------2</link>
            <guid isPermaLink="false">https://medium.com/p/98a563a916bf</guid>
            <category><![CDATA[webdataset]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[distributed-training]]></category>
            <category><![CDATA[aws-s3]]></category>
            <dc:creator><![CDATA[Ahmad Sachal]]></dc:creator>
            <pubDate>Mon, 22 May 2023 11:40:54 GMT</pubDate>
            <atom:updated>2023-05-22T11:40:54.273Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why I Chose WebDataset for Training on 50TB of Data?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*WvBa-iKRLdK4Ab4G.jpg" /><figcaption>Image Source: <a href="https://www.weka.io/learn/ai-ml/big-data-machine-learning/">Weka</a></figcaption></figure><p>I recently worked on a Distributed Training and MLOps project in which one of our goals was to make our training of a large model with huge data, highly scalable. This article will cover the data loading choices we made to achieve this.</p><h4>Understanding of the Problem</h4><p>Here are some of the points to help you understand the problem:</p><ul><li>50 TB of image data (LAION-5B) would be used for training.</li><li>Existing code was in PyTorch and for single GPU and single node setup.</li><li>AWS was the cloud service used. The existing system required all the training data to be loaded to EFS and then processed.</li><li>Consequently, 50TB of data on EFS called for high cost consumption and great storage requirements.</li><li>The DataLoader loaded small jpg files one by one which added additional processing overhead.</li></ul><h4>Existing Data Loading Method</h4><p>The existing DataLoader used a MapDataset to load all the files into memory and process it. This caused increased loading time and made parallelization limited and training less scalable.</p><p>For understanding the solutions implemented to alter this, we first need to understand the difference between Pytorch’s MapDataset and IterableDataset.</p><p><strong>MapDataset</strong></p><p>A PyTorch dataset that loads all the data into memory and applies a user-defined mapping function to each item in the dataset.</p><p>It implements the <em>__getitem__()</em> and <em>__len__()</em> protocols and represents a map from (possibly non-integral) indices/keys to data samples.</p><p>Here is a simple example code for a MapDataset data loader:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cNCvgIGsp_dpWurz" /><figcaption>Sample Code for MapDataset Data Loader — ChatGPT</figcaption></figure><p><strong>IterableDataset</strong></p><p>A PyTorch dataset that reads data on-the-fly and defines an __iter__() method to return an iterator that yields the data items.</p><p>It implements the <em>__iter__()</em> protocol and represents an iterable over data samples. This type of dataset is particularly suitable for cases where random reads are expensive or even improbable and where the batch size depends on the fetched data.</p><p>Here is a simple example code for an IterableDataset data loader:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9wNhAb-m6z4p7kTG" /><figcaption>Sample Code for IterableDataset DataLoader — ChatGPT</figcaption></figure><h3>Issues with Using MapDataset</h3><h4><strong>Memory Consumption</strong></h4><ol><li>Loads all the data into memory at once.</li><li>Can lead to high memory consumption.</li></ol><h4><strong>Slow Loading Times</strong></h4><ol><li>If your dataset is too large to fit into memory, you may need to load it from disk each time you access a sample.</li><li>This can result in slow loading times, which can slow down your training process and increase training time.</li></ol><h4><strong>I/O Bottleneck</strong></h4><ol><li>When working with large datasets, I/O (input/output) can become a bottleneck. MapDataset loads all the data into memory at once, which can result in a large number of I/O operations to read the data from disk.</li><li>This can slow down your program and reduce performance.</li></ol><h4><strong>Limited Parallelism</strong></h4><ol><li>MapDataset uses a single process to load data, which can limit the amount of parallelism you can achieve.</li><li>This can be a problem when working with large datasets, as parallelism can help speed up data loading and improve performance.</li></ol><h4><strong>Limited Scalability</strong></h4><ol><li>MapDataset may not be scalable to larger datasets or distributed environments. As the size of your dataset grows, it may become impractical to load all the data into memory at once.</li><li>In distributed environments, you may need to load data from multiple nodes or partitions, which can be challenging with a MapDataset.</li></ol><h3>Using WebDataset for Data Loading</h3><h4>TAR (.tar) File Format</h4><p>A tar file is a type of archive file that can contain multiple files within it.</p><p>The contents of the tar file are stored in a contiguous sequence of blocks.</p><p>Here is the order in which the files are compressed within the .tar file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2I9yzn9W_AYdQ1A_" /><figcaption>Without Directory Structure — ChatGPT</figcaption></figure><h4>WebDataset</h4><p><a href="https://github.com/webdataset/webdataset/blob/main/readme.ipynb">WebDataset</a> is a Python package that provides a data loading and processing pipeline for working with large-scale datasets in PyTorch.</p><p>It uses a sharded, shufflable TAR (.tar) file format that is efficient for networked and distributed storage. It can efficiently load large .tar files containing thousands of images while providing options to decode the images on the fly.</p><p>WebDataset implements standard PyTorch IterableDataset interface and works with the PyTorch DataLoader.</p><h4>DataLoader Using WebDataset</h4><p>WebDataset provides several important options for reading and processing the data via .tar files on the fly. Options like <em>decode</em> can also be used to decode the images to the required format on the fly just in one line.</p><p>Here is a simple example of a data loader with IterableDataset using WebDataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6Y4NBctrkyLDfQQ5" /><figcaption>Sample Code for IterableDataset Data Loader Using WebDataset — ChatGPT</figcaption></figure><h4>Streaming in the Data from S3 Directly with Pipe Method</h4><p>WebDataset provides a convenient way to stream in data from Amazon S3 by leveraging the pipe command. When data is requested from S3, it is returned in chunks, which can then be piped into the WebDataset pipeline for further processing.</p><p>Here’s how the URL should be revised and used for this method:</p><pre>url = &quot;s3://bucket/data.tar&quot;<br>s3_url = f&quot;pipe:aws s3 cp {url} -&quot;<br><br>dataset = wds.WebDataset(s3_url)</pre><p>This enables efficient loading and preprocessing of large datasets without the need to download the entire dataset to local storage, saving both time and storage space.</p><p>Additionally, WebDataset supports multi-threaded decoding, allowing for even faster data loading and preprocessing.</p><h4>Nodesplitter for Data Distribution</h4><p>WebDataset provides options to split data across nodes or workers with the help of <em>nodesplitter</em>. If there are 20 tar files (tar URLs) and 2 GPUs, nodesplitter will distribute all these URLs across the two GPUs so that the data is distributed and there is no duplication.</p><p>Here is how the data is split by worker in the code:</p><pre>dataset = wds.WebDataset(s3_url, nodesplitter=wds.split_by_worker)</pre><p>This distribution happens at the shard level, not at the sample level. If sample level distribution is needed, additional workarounds need to be implemented inside the data_generator method in the data loader.</p><p>The logic would roughly be something like this assuming images inside the .tar file are named as 1, 2, 3, … :</p><pre>num_workers = int(os.environ[&quot;WORLD_SIZE&quot;])<br>rank = int(os.environ[&#39;RANK&#39;])<br>img_obj = next(iter(iter_dataset))<br>img_val, img_id = img_obj<br>if int(img_id) % num_workers != rank:<br>  continue<br>else:<br>  yield img_val</pre><h4>How Is the Data Loader with WebDataset Better?</h4><ul><li>Better data loading performance as it allows for lazy-loading of data, which means that data is loaded on-demand as needed, rather than loading all data into memory upfront.<br>– This reduces memory usage and allows for larger datasets to be processed.</li><li>Support for shuffling of data without having to load all data into memory, which can be a challenge for large datasets.<br>– This is accomplished by shuffling the tar file shards before loading them, which ensures that the data is shuffled across all samples.</li><li>Allows for parallel processing of data, which can significantly speed up the data loading process.<br>– It achieves this by utilizing Python’s multiprocessing module to parallelize the data loading across multiple CPU cores.</li><li>Cost effective method for loading large amount of data from AWS S3.</li></ul><h4>Some of the Alternate Options</h4><ul><li>Use <strong>s3Dataset</strong> for IterableDataset with s3 and stream input data samples. (Lacks some options WebDataset provides and is said to be not fully compatible with Pytorch’s Distributed Training Framework)</li><li>Use <strong>FSx Lustre</strong> to load the data from s3 and use Sagemaker’s <strong>FastFile</strong> input mode to stream in data. (Too costly)</li><li>Use MapDataset or IterableDataset with Sagemaker’s <strong>FastFile</strong> mode directly. (Requires Sagemaker setup)</li></ul><h3>Conclusion</h3><p>When training on a huge amount of data, say 50TB of image data, the code setup should be highly scalable and efficient to achieve maximum speed and reduced costs.</p><p>For this, one of the key things is to choose how we load and input the data for training. This article explains how an IterableDataset can be a better data loading choice than a MapDataset we normally use. The article also covers how WebDataset assists in further improving the training performance.</p><p>The options chosen are based on the problem requirements and on what we thought to be the best options after experimenting. Feel free to share your knowledge in the responses or share any suggestions you may have on this topic. I’d be happy to learn from you :)</p><h3>For More on WebDataset</h3><ul><li><a href="https://github.com/webdataset/webdataset/blob/main/readme.ipynb">WebDataset Readme GitHub</a></li></ul><h3>Articles on Python Programming</h3><ul><li><a href="https://medium.com/red-buffer/python-production-level-coding-practices-4c39246e0233">Python: Production-Level Coding Practices</a></li><li><a href="https://medium.com/red-buffer/programming-methods-recursion-badd504af948">Programming Methods: Recursion</a></li></ul><h3>Articles on Data Science and Machine Learning</h3><ul><li><a href="https://medium.com/red-buffer/what-is-feature-engineering-5baf02f1b4f1">What Is Feature Engineering?</a></li><li><a href="https://medium.com/red-buffer/traffic-signs-recognition-cnn-model-with-tensorflow-serving-d2ed249c943e">Traffic Signs Recognition: CNN Model with Tensorflow Serving</a></li></ul><h3>Articles on Desktop App Development (Electronjs)</h3><ul><li><a href="https://medium.com/red-buffer/integrating-python-flask-backend-with-electron-nodejs-frontend-8ac621d13f72">Integrating Python Flask Backend with Electron (Nodejs) Frontend</a></li><li><a href="https://medium.com/red-buffer/electron-builder-packaging-electron-nodejs-application-along-with-flask-app-for-windows-fc26f5a29870">Electron Builder: Packaging Electron (Nodejs) Application (along with Flask app) for Windows</a></li><li><a href="https://medium.com/red-buffer/how-to-create-a-splash-screen-for-electron-app-602b4da406d">How to Create a Splash Screen for Electron App</a></li></ul><h3>About Ahmad Sachal</h3><ul><li>LinkedIn Profile — <a href="https://www.linkedin.com/in/ahmad-sachal/">linkedin.com/in/ahmad-sachal/</a></li></ul><h3>About Red Buffer</h3><ul><li>Website — <a href="https://redbuffer.ai/">redbuffer.ai</a></li><li>LinkedIn Page — <a href="https://www.linkedin.com/company/red-buffer">linkedin.com/company/red-buffer</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98a563a916bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/red-buffer/why-did-i-choose-webdataset-for-training-on-50tb-of-data-98a563a916bf">Why Did I Choose WebDataset for Training on 50TB of Data?</a> was originally published in <a href="https://medium.com/red-buffer">Red Buffer</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Programming Methods: Recursion]]></title>
            <link>https://medium.com/red-buffer/programming-methods-recursion-badd504af948?source=rss-85ddcd6488de------2</link>
            <guid isPermaLink="false">https://medium.com/p/badd504af948</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[recursion-python]]></category>
            <category><![CDATA[recursive-function]]></category>
            <dc:creator><![CDATA[Ahmad Sachal]]></dc:creator>
            <pubDate>Fri, 18 Mar 2022 10:53:09 GMT</pubDate>
            <atom:updated>2022-03-18T10:53:09.908Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RzsENqKDC_gE42PrCLrspQ.png" /></figure><p>In this article, we will cover a programming technique that is very complex and confusing in terms of implementation but very useful and simply brilliant.</p><h4>What Is Recursion?</h4><p>Recursion is a computer programming technique in which a function calls itself. If we have a recursive function A, then in recursion, we’d either have A call itself or it could call another function B that would then call A. It is either the recursive function directly calling itself or indirectly calling itself via another function like function B here. A simple recursive function looks as below:</p><pre>def func()<br>         |<br>         |<br>    func() //recursive call</pre><p>Now that we have explained what recursion is, we’ll go through some examples to further our understanding.</p><h3>Examples</h3><h4>Iterating without For Loop</h4><p>This is a simple example to explain how we can use recursion. We are defining a function that takes an integer as an argument and increments it by one till it is equal to 10. Normally, we use a <em>for</em> <em>loop</em> for this but we are able to do this with just a recursive function as shown below:</p><pre>def do_recursion(a):<br>  a += 1<br>  if a == 10:<br>    return a<br>  else:<br>    return do_recursion(a)</pre><pre>a = 1<br>print(do_recursion(a))</pre><p>Of course, this example is not a use case by itself though easy to understand, but even <em>if the </em>condition had been something else, we could just do recursion until that condition was met. This is useful when we need to recursively check for a condition if it has become true yet or not. If you are waiting for a process that takes an indefinite time to complete in order to go to the next process, you can use a recursive call with <em>if-else </em>statements.</p><p>We’ll now go through a more complex example.</p><h4>Factorial Function</h4><p>We want to create a function that takes an integer input and returns its factorial. Before going through this solution, you can brainstorm the logic that will go behind this function. It’s rather complex using loops but easy and efficient using recursion.</p><p>The factorial function will be as below:</p><pre><strong>def</strong> recursive_factorial(n):<br>  <strong>if</strong> n <strong>==</strong> 1:<br>    <strong>return</strong> n<br>  <strong>else</strong>:<br>    <strong>return</strong> n <strong>*</strong> recursive_factorial(n<strong>-</strong>1)</pre><p>We can see that we only need an if-else condition to make the recursion work. When <em>n=1</em>, we return <em>n</em>, otherwise we return <em>n*function(n-1).</em></p><p>Since there can be a few edge cases as well depending on the input integer, the implementation of this is as below:</p><pre>num <strong>=</strong> 10<br># placing checks for edge cases</pre><pre><strong>if</strong> num &lt; 0:<br>  <strong>print</strong>(&quot;Invalid input! Please enter a positive number.&quot;)<br><strong>elif</strong> num <strong>==</strong> 0:<br>  <strong>print</strong>(&quot;Factorial of number 0 is 1&quot;)<br><strong>else</strong>:<br>  print(&quot;Factorial of number %s is %s&quot;, (num,<br>        recursive_factorial(num)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*QkExhD-Q9cJPS0RlkSflNg.jpeg" /><figcaption>factorial(n-1) again</figcaption></figure><p>P.S: There are plenty of recursion memes. Do comment if you’ve got a good one.</p><h3>Conclusion</h3><p>Recursion, even if it is sometimes hard to follow, is very useful when we need to make a complex task easier. It is especially useful when your logic needs to go through multilevel iterations e.g. while traversing through nested lists or dictionaries. <a href="https://realpython.com/python-recursion/">Here</a>’s a detailed tutorial on recursion that properly demonstrates the difference between recursive and non-recursive approaches for some use cases.</p><h3>Articles on Programming</h3><ul><li><a href="https://medium.com/red-buffer/python-production-level-coding-practices-4c39246e0233">Python: Production-Level Coding Practices</a></li></ul><h3>Articles on Data Science and Machine Learning</h3><ul><li><a href="https://medium.com/red-buffer/what-is-feature-engineering-5baf02f1b4f1">What Is Feature Engineering?</a></li><li><a href="https://medium.com/red-buffer/traffic-signs-recognition-cnn-model-with-tensorflow-serving-d2ed249c943e">Traffic Signs Recognition: CNN Model with Tensorflow Serving</a></li></ul><h3>About Ahmad Sachal</h3><ul><li>LinkedIn Profile — <a href="https://www.linkedin.com/in/ahmad-sachal/">linkedin.com/in/ahmad-sachal/</a></li></ul><h3>About Red Buffer</h3><ul><li>Website — <a href="http://redbuffer.net/">redbuffer.net</a></li><li>LinkedIn Page — <a href="https://www.linkedin.com/company/red-buffer">linkedin.com/company/red-buffer</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=badd504af948" width="1" height="1" alt=""><hr><p><a href="https://medium.com/red-buffer/programming-methods-recursion-badd504af948">Programming Methods: Recursion</a> was originally published in <a href="https://medium.com/red-buffer">Red Buffer</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Create a Splash Screen for Electron App]]></title>
            <link>https://medium.com/red-buffer/how-to-create-a-splash-screen-for-electron-app-602b4da406d?source=rss-85ddcd6488de------2</link>
            <guid isPermaLink="false">https://medium.com/p/602b4da406d</guid>
            <category><![CDATA[website-design]]></category>
            <category><![CDATA[splash-screen]]></category>
            <category><![CDATA[electronjs]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[nodejs]]></category>
            <dc:creator><![CDATA[Ahmad Sachal]]></dc:creator>
            <pubDate>Fri, 12 Nov 2021 08:22:18 GMT</pubDate>
            <atom:updated>2021-11-12T08:22:18.997Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/664/0*5aTJRdOcQBmQFrXV.jpg" /></figure><p>This is a short tutorial on how to create a splash screen for an <em>electron </em>application.</p><h4>What is a Splash Screen?</h4><p>A splash screen is an introductory window that appears when we launch an application. When the application components are ready, the splash screen disappears and the main application window appears. For example, when you launch Twitter, first a blue screen with the bird appears and then your app timeline shows up.</p><h4>Step 1: Clone the Electron App Repository</h4><p>The first thing to do is go to your project directory and clone the quickstart electron app directory in the terminal with the following commands given on <a href="https://github.com/electron/electron-quick-start/blob/master/README.md">GitHub</a>.</p><pre># Clone this repository<br>git clone <a href="https://github.com/electron/electron-quick-start">https://github.com/electron/electron-quick-start</a></pre><pre># Go into the repository<br>cd electron-quick-start</pre><pre># Install dependencies<br>npm install</pre><pre># Run the app<br>npm start</pre><p>On <em>npm start</em>, you’ll get the following app screen:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*WRBL1DiLGlojmVXiYDF3lw.png" /></figure><p>But, we don’t want just this. We want a splash screen that appears before this main window loads. For that, we need a splash screen HTML file that we can direct the app to when it starts.</p><h4>Step 2: Create an HTML File for the Splash Screen</h4><p>All we need for this is a simple HTML file that would show a loader and a background if needed. Here’s the code for our simple splash screen with a loader with some CSS:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcodepen.io%2Fahmadsachal%2Fembed%2Fpreview%2FgOxKwEz%3Fdefault-tabs%3Dcss%252Cresult%26height%3D600%26host%3Dhttps%253A%252F%252Fcodepen.io%26slug-hash%3DgOxKwEz&amp;display_name=CodePen&amp;url=https%3A%2F%2Fcodepen.io%2Fahmadsachal%2Fpen%2FgOxKwEz&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=codepen" width="800" height="600" frameborder="0" scrolling="no"><a href="https://medium.com/media/28537fbb8747b4a9f22a5c433f8bf535/href">https://medium.com/media/28537fbb8747b4a9f22a5c433f8bf535/href</a></iframe><h4>Step 3: Create the Splash Screen Window in main.js</h4><p>Now that we have our <em>splash.html </em>ready, all we need to do is create the splash screen window in the electron app’s main.js file. The function we are concerned within the code is the <em>createWindow </em>function which looks as below:</p><pre>function createWindow() {<br>  // Create the browser window.<br>  const mainWindow = new BrowserWindow({<br>    width: 800,<br>    height: 600,<br>    webPreferences: {<br>       preload: path.join(__dirname, &#39;preload.js&#39;)<br>     }<br>  })</pre><pre>  // and load the index.html of the app.<br>  mainWindow.loadFile(&#39;index.html&#39;)<br>  mainWindow.center();</pre><pre>  // Open the DevTools.  <br> // mainWindow.webContents.openDevTools()<br>}</pre><p>We first need to hide the main window we’re creating above by setting the <em>show </em>property to <em>false</em>, so the mainwindow function looks like this:</p><pre>const mainWindow = new BrowserWindow({<br>  width: 800,<br>  height: 600,<br>  <strong>show: false,</strong><br>  webPreferences: {<br>    preload: path.join(__dirname, &#39;preload.js&#39;)<br>  }<br>})</pre><p>Now we create the splash screen window just like we created the mainwindow.</p><pre>var splash = new BrowserWindow({<br>     width: 500, <br>     height: 300, <br>     transparent: true, <br>     frame: false, <br>     alwaysOnTop: true <br>});</pre><p>We load the splash.html file in this window using:</p><pre>splash.loadFile(&#39;splash.html&#39;);<br>splash.center();</pre><p>So we have successfully managed to create our splash screen and on starting the application, we’ll get the splash screen loaded first. But we still need to show our main window after the splash screen.</p><h4>Step 4: Show the Main Window When Your App Is Ready</h4><p>This is actually the part where creating the splash screen is very beneficial. In our case, we don’t currently have to start some server or spawn another application on starting the app, but that is not always the case. It is wrong to show the main window when the app is not actually ready. In my experience, I had to spawn a flask application and then send requests to it via the electron UI. If the flask app hadn’t started (since it can take a few seconds), I’d get an error on sending a post request.</p><p>The best thing to do is show the splash screen and wait in the background for the components to be fully loaded and then, when the condition is satisfied, close the splash window and show the main window. It’s very simple to do so. We just need to write the following after our conditions are met:</p><pre>splash.close();<br>mainWindow.show();</pre><p>Since, for this tutorial, there isn’t any background process to wait for to load, we’ll just wait for 5–10 seconds and then show our main window using <em>setTimeout().</em></p><pre>setTimeout(function () {<br>  splash.close();<br>  mainWindow.show();<br>}, 5000);</pre><p>Here’s how it looks (changed the screen background a bit):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*DRI66VBAf4TkPay_2qfAqw.gif" /></figure><p>After 5 seconds (or whatever time you set), the splash window closes and the main window appears.</p><p>The code can be found <a href="https://github.com/ahmadsachal/electron_splash_screen.git">here</a>.</p><h3>Previous Articles on Electron</h3><ul><li><a href="https://medium.com/red-buffer/integrating-python-flask-backend-with-electron-nodejs-frontend-8ac621d13f72">Integrating Python Flask Backend with Electron (Nodejs) Frontend</a></li><li><a href="https://medium.com/red-buffer/electron-builder-packaging-electron-nodejs-application-along-with-flask-app-for-windows-fc26f5a29870">Electron Builder: Packaging Electron (Nodejs) Application (along with Flask app) for Windows</a></li></ul><h3>About Ahmad Sachal</h3><ul><li>LinkedIn Profile — <a href="https://www.linkedin.com/in/ahmad-sachal/">linkedin.com/in/ahmad-sachal/</a></li></ul><h3>About Red Buffer</h3><ul><li>Website — <a href="http://redbuffer.net/">redbuffer.net</a></li><li>LinkedIn Page — <a href="https://www.linkedin.com/company/red-buffer">linkedin.com/company/red-buffer</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=602b4da406d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/red-buffer/how-to-create-a-splash-screen-for-electron-app-602b4da406d">How to Create a Splash Screen for Electron App</a> was originally published in <a href="https://medium.com/red-buffer">Red Buffer</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Traffic Signs Recognition: CNN Model with Tensorflow Serving]]></title>
            <link>https://medium.com/red-buffer/traffic-signs-recognition-cnn-model-with-tensorflow-serving-d2ed249c943e?source=rss-85ddcd6488de------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2ed249c943e</guid>
            <category><![CDATA[tensorflow-serving]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Ahmad Sachal]]></dc:creator>
            <pubDate>Mon, 01 Nov 2021 07:09:17 GMT</pubDate>
            <atom:updated>2021-11-01T07:09:17.628Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*Hn5OarBg2Z2Ju0pg.jpeg" /></figure><p>In this article, we will do a Traffic Signs Recognition project with a Kaggle dataset of German traffic signs, which would require deep learning, and we’ll serve our model and test it out using Tensorflow Serving. We hit about 97% training accuracy here.</p><h4>About the Dataset</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/0*LFS88CXIGOP84TmO.jpg" /><figcaption>Nope. Not this kind of signs…</figcaption></figure><p>The traffic signs dataset we’re gonna work with can be downloaded from Kaggle <a href="https://www.kaggle.com/meowmeowmeowmeowmeow/gtsrb-german-traffic-sign">here</a>. It contains images of 43 classes of traffic signs. Each class has a separate folder and a lot of training images within those folders. We have a similar testing dataset as well and have some additional images that we could use as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/170/1*Yy6KI2E-8TRF7z_yHvSiiw.png" /><figcaption>German Traffic Sign Recognition Dataset on Kaggle</figcaption></figure><h4>Retrieving Data Images and Labels</h4><p>So, to read the images and get their data, we can either use the computer vision library <em>cv2</em> (<a href="https://pypi.org/project/opencv-python/">opencv-python</a>) or we could go for <a href="https://pypi.org/project/Pillow/"><em>Pillow</em></a><em>. </em>I’ve gone with cv2 here and you can load an image as:</p><pre>import cv2<br>image = cv2.imread(path)</pre><p>Now, we want to load images for all the classes present in different folders, scale them to a standard size, and turn them into arrays of data and labels, which is done as below:</p><pre>import glob<br>import numpy as np<br>import cv2</pre><pre>data = []<br>labels = []<br>classes = 43<br>cur_path = &#39;/path/to/your/dataset/directory/&#39;</pre><pre>for i in range(classes):<br>    path = cur_path + &#39;Train/&#39; + str(i) + &#39;/*&#39;<br>    images = glob.glob(path)<br>    for im in images:<br>        try:<br>            image = cv2.imread(im)<br>            image = cv2.resize(image, (30,30))<br>            data.append(image)<br>            labels.append(i)<br>        except:<br>            print(&quot;Error loading image&quot;)</pre><pre>data = np.array(data)<br>labels = np.array(labels)</pre><p>Since the data is now loaded in the form of arrays, we can now perform our train test split:</p><pre>from sklearn.model_selection import train_test_split</pre><pre>X_train, X_test, y_train, y_test = train_test_split(data, labels, test_size=0.2, random_state=42)</pre><p>We need a categorical encoding for our labels so we do one-hot encoding on our label data.</p><pre>from tensorflow.keras.utils import to_categorical</pre><pre>y_train = to_categorical(y_train, 43)<br>y_test = to_categorical(y_test, 43)</pre><h3>Building the Model</h3><h4>Model Architecture</h4><p>To efficiently build and train our model, we need to work with convolutional neural networks. The model architecture used is from Data Flair’s great work where they achieved 95% accuracy with this <a href="https://data-flair.training/blogs/python-project-traffic-signs-recognition/">here</a>. The difference in accuracies seems to be because of the usage of cv2 instead of pillow (have to test out why). The architecture is as below:</p><pre>from tensorflow.keras.models import Sequential, load_model<br>from tensorflow.keras.layers import Conv2D, MaxPool2D, Dense, Flatten, Dropout</pre><pre>model = Sequential()<br>model.add(Conv2D(filters=32, kernel_size=(5,5), activation=&#39;relu&#39;, input_shape=X_train.shape[1:]))<br>model.add(Conv2D(filters=32, kernel_size=(5,5), activation=&#39;relu&#39;))<br>model.add(MaxPool2D(pool_size=(2, 2)))<br>model.add(Dropout(rate=0.25))<br>model.add(Conv2D(filters=64, kernel_size=(3, 3), activation=&#39;relu&#39;))<br>model.add(Conv2D(filters=64, kernel_size=(3, 3), activation=&#39;relu&#39;))<br>model.add(MaxPool2D(pool_size=(2, 2)))<br>model.add(Dropout(rate=0.25))<br>model.add(Flatten())<br>model.add(Dense(256, activation=&#39;relu&#39;))<br>model.add(Dropout(rate=0.5))<br>model.add(Dense(43, activation=&#39;softmax&#39;))</pre><h4>Brief Description of the Layers</h4><p>We have a sequential model and for our image array data, we are using <strong><em>Conv2D</em></strong> layers with filters of 32. <em>Filters</em> in the first layer detect simple features and edges. As the architecture goes, the filters in the deeper layers detect more complex features that get us to our goal.</p><p>We are using <em>ReLU</em> as the activation function in the Conv2D layers. It basically is a piecewise linear function that outputs the value when it’s positive and returns zero if it’s negative. In our dense layer, we are using <em>Softmax </em>which turns our output to the probability of membership of each class and, combined, sums to 1.</p><p><em>MaxPool2D</em> layer is the pooling layer used in the deep neural network. It finds and returns the maximum value for each feature map.</p><h4>Compiling the Model</h4><p>Next, we have to compile the model and for this, we need an ideal optimizer and a loss function. Here, we go with Adam optimizer and Categorical Crossentropy Loss as below:</p><pre>model.compile(loss=&#39;categorical_crossentropy&#39;, optimizer=&#39;adam&#39;, metrics=[&#39;accuracy&#39;])</pre><p>Adam optimizer is an adaptive optimization algorithm that updates the learning rate based on moments.</p><p>The Categorical Crossentropy Loss function is used for multiclass problems. The mathematical formula for the loss function is as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*UAUvkZQyNYzm9ifhc2_Ahg.png" /></figure><h4>Fitting the Model</h4><p>Now we fit our model by providing the training and validation data, batch size, and epochs. For me, the best number of epochs is 18. Our accuracy becomes stable between 15 and 20 epochs, so we are choosing the number where we are getting the minimum training and validation losses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NcxlKecRag8xbUC6TLvAAQ.png" /></figure><p>Batch size is the number of samples before the model is updated. We are keeping it at 64 as it gives good results, but you can test this out further with your model. The <em>model.fit</em> code is as below:</p><pre>history = model.fit(X_train, y_train, batch_size=64, epochs=18, validation_data=(X_test, y_test))</pre><h3>Serving the Model with Tensorflow Serving</h3><h4>Saving the Model with Tensorflow Keras’s save_model</h4><p>In order to serve the model with TensorFlow serving, we need the model to be saved in the required <em>.pb</em> format. For this we just save our model with our required configurations by using <em>tf.keras.models.save_model </em>as given below:</p><pre>MODEL_DIR = &#39;/save/model/directory/&#39;<br>version = 1<br>export_path = os.path.join(MODEL_DIR, str(version))<br>tf.keras.models.save_model(<br>    model,<br>    export_path,<br>    overwrite=True,<br>    include_optimizer=True,<br>    save_format=None,<br>    signatures=None,<br>    options=None<br>)</pre><p>You can then check the saved model by:</p><pre>!saved_model_cli show --dir {export_path} --all</pre><h4>Install Tensorflow Model Server on Colab or Linux</h4><p>If you are using Colab or Linux, then TensorFlow serving’s own guide recommends installing it as given below:</p><pre>import sys<br># We need sudo prefix if not on a Google Colab.<br>if &#39;google.colab&#39; not in sys.modules:<br>  SUDO_IF_NEEDED = &#39;sudo&#39;<br>else:<br>  SUDO_IF_NEEDED = &#39;&#39;</pre><pre>!echo &quot;deb <a href="http://storage.googleapis.com/tensorflow-serving-apt">http://storage.googleapis.com/tensorflow-serving-apt</a> stable tensorflow-model-server tensorflow-model-server-universal&quot; | {SUDO_IF_NEEDED} tee /etc/apt/sources.list.d/tensorflow-serving.list &amp;&amp; \<br>curl <a href="https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg">https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg</a> | {SUDO_IF_NEEDED} apt-key add -<br>!{SUDO_IF_NEEDED} apt update</pre><pre>!{SUDO_IF_NEEDED} apt-get install tensorflow-model-server</pre><p>Tensorflow serving is not available on Windows. I use Windows and don’t use Colab so I went for WSL (Windows Subsystem for Linux) Ubuntu for my purpose. You can simply install TensorFlow-model-server with the above lines on WSL.</p><h4>Serve the Model</h4><p>Once you have installed the model server, it’s pretty easy to serve it now. You already defined the version folder of your model while saving it with <em>save_model. </em>So now you need to give the following command in your command line to serve the model:</p><pre>tensorflow_model_server <br>      --port=8500 --rest_api_port=8598 <br>      --model_name=traffic_signs_model <br>      --model_base_path=/mnt/&lt;drive_folder&gt;/&lt;model_directory&gt;/</pre><p>You can see that we need our hosting port and a rest API port, we need to name our model and then provide the model base path.</p><p>The <em>/mnt/ </em>is basically used in WSL to access the Windows directories. You can mention your model directory as normally done if you are not on WSL.</p><p>Note that you do not need to mention the version since TensorFlow serving will automatically pick the version.</p><h3>Model Inferencing</h3><h4>Name Classes</h4><p>For proper inferences, we need to name the classes or what those labels represent. So we make a dictionary of classes.</p><pre>#dictionary to label all traffic signs class.<br>classes = { 0:&#39;Speed limit (20km/h)&#39;,<br>            1:&#39;Speed limit (30km/h)&#39;, <br>            2:&#39;Speed limit (50km/h)&#39;, <br>            3:&#39;Speed limit (60km/h)&#39;, <br>            4:&#39;Speed limit (70km/h)&#39;, <br>            5:&#39;Speed limit (80km/h)&#39;, <br>            6:&#39;End of speed limit (80km/h)&#39;, <br>            7:&#39;Speed limit (100km/h)&#39;, <br>            8:&#39;Speed limit (120km/h)&#39;, <br>            9:&#39;No passing&#39;, <br>            10:&#39;No passing veh over 3.5 tons&#39;, <br>            11:&#39;Right-of-way at intersection&#39;, <br>            12:&#39;Priority road&#39;, <br>            13:&#39;Yield&#39;, <br>            14:&#39;Stop&#39;, <br>            15:&#39;No vehicles&#39;, <br>            16:&#39;Veh &gt; 3.5 tons prohibited&#39;, <br>            17:&#39;No entry&#39;, <br>            18:&#39;General caution&#39;, <br>            19:&#39;Dangerous curve left&#39;, <br>            20:&#39;Dangerous curve right&#39;, <br>            21:&#39;Double curve&#39;, <br>            22:&#39;Bumpy road&#39;, <br>            23:&#39;Slippery road&#39;, <br>            24:&#39;Road narrows on the right&#39;, <br>            25:&#39;Road work&#39;, <br>            26:&#39;Traffic signals&#39;, <br>            27:&#39;Pedestrians&#39;, <br>            28:&#39;Children crossing&#39;, <br>            29:&#39;Bicycles crossing&#39;, <br>            30:&#39;Beware of ice/snow&#39;,<br>            31:&#39;Wild animals crossing&#39;, <br>            32:&#39;End speed + passing limits&#39;, <br>            33:&#39;Turn right ahead&#39;, <br>            34:&#39;Turn left ahead&#39;, <br>            35:&#39;Ahead only&#39;, <br>            36:&#39;Go straight or right&#39;, <br>            37:&#39;Go straight or left&#39;, <br>            38:&#39;Keep right&#39;, <br>            39:&#39;Keep left&#39;, <br>            40:&#39;Roundabout mandatory&#39;, <br>            41:&#39;End of no passing&#39;, <br>            42:&#39;End no passing veh &gt; 3.5 tons&#39; }</pre><h4>Load Test Images</h4><p>Now we’ll load a small batch of test images. We’ll read the first five images from our test images directory and append them to a list that we can send for our predictions.</p><pre>data = []<br>labels = []<br>cur_path = &#39;/dataset/directory/&#39;</pre><pre>path = cur_path + &#39;Test/*&#39;<br>images = glob.glob(path)</pre><pre>#Retrieving the images and their labels<br>for im in images[:5]:<br>    try:<br>        image = cv2.imread(im)<br>        image = cv2.resize(image, (30,30))<br>        data.append(image)<br>    except:<br>        print(&quot;Error loading image&quot;)</pre><pre>#Converting lists into numpy arrays<br>test_imgs = np.array(data)</pre><h4>Send Prediction Request to the Served Model</h4><p>Now we create a JSON object with which we send our post request to the model directory. Define the headers as below, and send the test images as type <em>list</em> in the JSON object.</p><pre>data = json.dumps({&quot;signature_name&quot;: &quot;serving_default&quot;,<br>                   &quot;inputs&quot;: test_imgs.tolist()})</pre><pre>headers = {&quot;content-type&quot;: &quot;application/json&quot;}</pre><p>Make the post request to the model as below and you’ll receive a response immediately from the model which you can then be interpreted.</p><pre>json_response = requests.post(<br>&#39;<a href="http://127.0.0.1:8598/v1/models/traffic_signs_model/versions/1:predict&#39;">http://127.0.0.1:8598/v1/models/traffic_signs_model/versions/1:predict&#39;</a>, data=data, headers=headers)</pre><pre>predictions = json.loads(json_response.text)[&#39;outputs&#39;]</pre><h4>Verify the Predictions</h4><p>Now, I’ll just verify the predictions for our five test images. We can do this by using our classes dictionary defined before and argmax on the predicted probabilities with:</p><pre>for pred in predictions:    <br>    print(np.argmax(pred), classes[np.argmax(pred)])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/1*ZPpktIZmwrvFtrzOe_LMEA.png" /></figure><p>We can visualize this better by displaying and matching their identified class as below:</p><pre>for pred, img in zip(predictions, test_imgs):<br>    plt.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))<br>    plt.title(&#39;Class &#39; + str(np.argmax(pred)) + &#39; : &#39;     <br>               + classes[np.argmax(pred)])<br>    plt.show()</pre><p>This will give us our 5 images with their label IDs as titles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*6vJi11f2xg-qH3-lyAKPPw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*BiuhWIHlSiiSweO6iTRRyg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*kTZkwZVIHwKnT4DM2MEsaQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*i9aEEbSE2Jhxj4_tNPyNWQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/257/1*k98k8qgBtQ2dOSr8_jOaEw.png" /></figure><p>Seems pretty accurate…</p><p>This is just to check inference for the first five images. We can make a post request with all the test images, save the prediction argmax for each image to a list and make use of the already labeled test image class ids given in the <em>Test.csv</em> file to compare the accuracy score with:</p><pre>from sklearn.metrics import accuracy_score</pre><pre>print(accuracy_score(labels, predicted_classes_list))</pre><p>Complete code can be found on <a href="https://github.com/ahmadsachal/traffic_signs_cnn_tfserving/blob/master/traffic_signs_cnn_tfserving.ipynb">Github</a>.</p><h3>References for More Learning</h3><ul><li><a href="https://www.tensorflow.org/tfx/tutorials/serving/rest_simple">Tensorflow Serving Guide</a></li><li><a href="https://www.tensorflow.org/tutorials/images/cnn">Convolutional Neural Network (CNN)</a></li><li><a href="https://www.analyticsvidhya.com/blog/2020/03/tensorflow-serving-deploy-deep-learning-models/">Deploying Multiple Models Via Tensorflow Serving</a></li></ul><h3>Previous Article</h3><ul><li><a href="https://medium.com/red-buffer/what-is-feature-engineering-5baf02f1b4f1">What Is Feature Engineering?</a></li></ul><h3>About Ahmad Sachal</h3><ul><li>LinkedIn Profile — <a href="https://www.linkedin.com/in/ahmad-sachal/">linkedin.com/in/ahmad-sachal/</a></li></ul><h3>About Red Buffer</h3><ul><li>Website — <a href="http://redbuffer.net/">redbuffer.net</a></li><li>LinkedIn Page — <a href="https://www.linkedin.com/company/red-buffer">linkedin.com/company/red-buffer</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2ed249c943e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/red-buffer/traffic-signs-recognition-cnn-model-with-tensorflow-serving-d2ed249c943e">Traffic Signs Recognition: CNN Model with Tensorflow Serving</a> was originally published in <a href="https://medium.com/red-buffer">Red Buffer</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Feature Engineering?]]></title>
            <link>https://medium.com/red-buffer/what-is-feature-engineering-5baf02f1b4f1?source=rss-85ddcd6488de------2</link>
            <guid isPermaLink="false">https://medium.com/p/5baf02f1b4f1</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <dc:creator><![CDATA[Ahmad Sachal]]></dc:creator>
            <pubDate>Mon, 06 Sep 2021 07:07:57 GMT</pubDate>
            <atom:updated>2021-09-06T07:07:57.136Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3u8u-8D1dgJo6Wjb.png" /></figure><p>In this article, we’ll look at what <em>Feature Engineering </em>is and discuss some techniques used for feature engineering.</p><h4>Feature</h4><p>In machine learning and data science, a <em>feature </em>is an individual measurable property or characteristic of a phenomenon, usually in the form of structured columns. For instance, we have a simple data table with students and their math exam marks as given below:</p><pre>          Name  Maths<br>  0   Chandler     71<br>  1   Ross         77<br>  2   Monica       91</pre><p>The columns i.e., <em>Name </em>and <em>Maths </em>are features in the data.</p><h4>What Is Feature Engineering?</h4><p>Feature Engineering is the step in data science in which you extract features from raw data to facilitate data analysis or machine learning.</p><p>When the goal is data analysis or machine learning, you need data that does not contain outliers, missing values, or unneeded features. Moreover, you need data that can be analyzed. If you want to plot graphs, you need to trim and scale your data accordingly. In machine learning, you need to engineer the data to make it compatible/standardized so that you can design an efficient machine learning model.</p><h3>Feature Engineering Techniques</h3><p>We will briefly discuss various techniques that are commonly used for feature engineering.</p><h4>Imputation — Dealing with Missing Values</h4><p>When we are working with data, we get a lot of null or missing values. These missing values must always be taken seriously since they can affect the analysis quite a lot and can pretty much dampen the performance of our machine learning models. There are a few ways to cater to this problem.</p><p><strong>Dropping Rows with Missing Values:</strong> We can drop the rows containing missing values. It is important, though, that we look and identify if the features we’re basing our analysis or model on have missing values or not, since we don’t want to drop significant data due to missing values of features that aren’t even going to be much significant for us. We can then drop the rows for those missing values. This isn’t the best method to deal with the problem because, in some cases, it could really reduce the data size and do more harm than good. So, you have the method of imputation to do a much better job for you here.</p><p>We have two types of imputation methods depending on the type of data.</p><p><strong>Numerical Imputation — </strong>For missing values of numerical data, we can either change the values to 0 or set it to the mean value of the columns as:</p><pre>data = data.fillna(0) #Replace NA value with 0</pre><pre>data = data.fillna(data.mean()) #Replace NA value with the mean value</pre><p><strong>Categorical Imputation — </strong>For missing values of categorical data, we can change the value to the most commonly occurring value or change it to “other”.</p><pre>data[&#39;column&#39;].fillna(data[&#39;column&#39;].value_counts().idxmax(), inplace=True)</pre><h4>One-Hot Encoding</h4><p>This is a feature engineering method to convert categorical data into a numeric form and is one of the most common ones used in machine learning. We cannot always work with categorical data directly for machine learning, in fact in most cases we can’t. We need the categorical data converted into the numeric form so that we could implement our algorithms efficiently.</p><p>In one-hot encoding, we convert our categorical data column into multiple columns of ones and zeros. For example, if we have a column named “colors” containing three values i.e., [‘<em>red’, ‘green’, ‘blue’]</em>, after the encoding method, we’d get:</p><pre>red, green, blue<br>  1,     0,    0<br>  0,     1,    0<br>  0,     0,    1</pre><p>Sklearn provides us a class for One-Hot Encoding and it can be imported and applied to the data with just one line. Documentation can be seen <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html">here</a>.</p><h4>Feature Scaling — Normalization and Standardization</h4><p>In machine learning, we need the features to have comparable value ranges. For example, we can not compare values of height and weight since they have different value ranges and completely different units, but we can scale the variables in a way that would make it easier to compare them and use the data efficiently. Following are the two common methods for feature scaling:</p><p><strong>Normalization</strong></p><p>In normalization, we scale all the features into a fixed range from 0 to 1. The formula used for normalization is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/168/0*vZ4vmhTc9Epg1hdK.png" /></figure><p>This method is also called min-max normalization.</p><p><strong>Standardization</strong></p><p>In standardization, values are centered around the mean value with unit variance. The formula is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/82/0*SX6gfvT0kaSXPZtQ.png" /></figure><p><em>where μ is mean and σ is the standard deviation</em></p><h4><strong>Handle Outliers</strong></h4><p>Outliers are the values that are either too high or too low relative to the overall data. Such values affect the skewness of the data and also affect the model prediction. It is, therefore, very important to handle outliers. Some of the methods are as follows:</p><ul><li><strong>Remove the values:</strong> One way of dealing with outliers is removing them from the overall distribution, but this method is not always the most efficient one since important data could be lost doing this.</li><li><strong>Imputation: </strong>We could just treat the outlier values like the missing values and do imputations as discussed before.</li><li><strong>Capping the max/min values: </strong>In this method, we can replace the max/min values with some arbitrary max/min values that would indicate that above or below that value, the variable will behave in a similar way.</li></ul><h4><strong>Log Transform</strong></h4><p>In order to handle skewed data (to make it more normal), we use logarithmic transformation. This method has more than one advantage as this also reduces our outlier problem by reducing the effect an outlier creates.</p><p>The important thing to keep in mind is that this method cannot be implemented directly on negative-valued data since that’ll throw an error. So either a different approach should be used or 1 could be added to change the data values to positive i.e., log(x+1).</p><h4><strong>Feature Split</strong></h4><p>Splitting data logically is a good way to extract more useful information from the available data. The implementation of the split is subject to the available data and the information it can provide. For a simple example, let’s say we have data with full names of people and their ages as below:</p><pre>Name                     Age<br>Mr. Rob Stark             25<br>Mr. Balian de Ibelin      30<br>Miss Mary Jane            23<br>Mrs. Caitylin Stark       50</pre><p>In the above data, you can see that genders are not mentioned, but we can extract this information by titles provided in the names i.e., <em>Mr, Mrs, etc.</em></p><p>You can string split the title part into a new column and categorize the available data to genders. This way you now can provide the model new information.</p><p>Apart from feature splitting, grouping methods are also used for feature extraction and get better results.</p><h3>Conclusion</h3><p>Whether we are to do analysis on data or want to make a good prediction model, we need to do feature engineering. It is the main step toward getting good results and usually takes a lot longer than the actual machine learning or analysis. It is, therefore, key to becoming well-versed in the techniques used for feature engineering. The article is not an exhaustive list of all the feature engineering techniques but is of the most common and useful ones. The main thing to focus on is that we have to extract/create/engineer the most significant and compatible features from raw data in accordance with the required task.</p><h3>Previous Article</h3><ul><li><a href="https://medium.com/red-buffer/python-production-level-coding-practices-4c39246e0233">Python: Production-Level Coding Practices</a></li></ul><h3>About Ahmad Sachal</h3><ul><li>LinkedIn Profile — <a href="https://www.linkedin.com/in/ahmad-sachal/">linkedin.com/in/ahmad-sachal/</a></li></ul><h3>About Red Buffer</h3><ul><li>Website — <a href="http://redbuffer.net/">redbuffer.net</a></li><li>LinkedIn Page — <a href="https://www.linkedin.com/company/red-buffer">linkedin.com/company/red-buffer</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5baf02f1b4f1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/red-buffer/what-is-feature-engineering-5baf02f1b4f1">What is Feature Engineering?</a> was originally published in <a href="https://medium.com/red-buffer">Red Buffer</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python: Production-Level Coding Practices]]></title>
            <link>https://medium.com/red-buffer/python-production-level-coding-practices-4c39246e0233?source=rss-85ddcd6488de------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c39246e0233</guid>
            <category><![CDATA[production]]></category>
            <category><![CDATA[good-coding-practice]]></category>
            <category><![CDATA[best-practices]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Ahmad Sachal]]></dc:creator>
            <pubDate>Mon, 05 Jul 2021 07:49:47 GMT</pubDate>
            <atom:updated>2021-07-05T07:49:47.005Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tKhIpBdPe172fFJeWvLeEw.jpeg" /></figure><p>The purpose of this article is to share what I have learned while working on production-level projects. I am keeping it Python-centric but the same practices apply to all coding languages and generally in all software development work. When I started working, there were a lot of flaws in my own coding style and a lot of inefficient coding habits, but I was fortunate enough to receive thorough guidance and learning from highly experienced fellow engineers, who taught me how to structure my code logically. So, I aim to share my learning in this piece and focus on how to do production-level coding in Python that is more flexible, dynamic, easier to make adjustments to, and reusable for future work.</p><h3>Break down the code</h3><blockquote>Always limit tasks to their specific functions.</blockquote><p>The software development principle you need to follow is the Separation of Concerns (SoC), which means you have to divide the code into distinct sections such that each section addresses its own separate concern. For instance, if you are writing a program to do addition on lists and then finding the square of each element of the list. It should be broken down into two functions/subtasks: 1. addition; 2. find square; (I know I am going with a very basic and boring example but it’s to facilitate understanding). Both these tasks should be separate functions which are later called to calculate the output.</p><pre>listA = [1, 2, 3, 4, 5]<br>listB = [9, 8, 7, 6, 5]</pre><pre>def find_sum(listA, listB):<br>  summed_list = np.add(listA, listB)<br>  return summed_list</pre><pre>def find_square(my_list):<br>  squared_list = np.square(my_list)<br>  return squared_list</pre><pre>summed_list = find_sum(listA, listB);<br>squared_list = find_square(summed_list)</pre><p>The main sense in doing this is that you are making functions that are specific to your own tasks. Let’s say there are now two new lists of which you need to find square roots after their sum. It will take unnecessary time in writing the whole code again separately for addition + finding the square root. You could instead just reuse the addition function and make a separate square root function to perform this task.</p><pre>def find_sqrt(my_list):<br>  sqrt_list = np.sqrt(my_list)<br>  return sqrt_list</pre><pre>new_summed_list = find_sum(listC, listD);<br>sqrt_list = find_sqrt(new_summed_list)</pre><p>If you were to do the whole thing without using functions, there is no chance of reusability for even a slightly different task. And if you do the first task in one single function (with addition and square), then you can’t reuse that function again (for just addition). This is why the task should be broken down into separate tasks even for smaller steps like element-wise addition or squaring in order to increase reusability and saving time in the long run.</p><h4>Advantages</h4><ul><li>Your program is reusable and you don’t have to waste extra time rewriting your code.</li><li>If there is a change in one of the functions, say Addition Function, you will only need to make the change in that function and not in all the programs wherever the addition is done. If you had an Addition/SquareRoot function and an Addition/Squaring function…the change would be made on both.</li></ul><h3>Start Using a Proper IDE</h3><blockquote>Using an IDE (Integrated Development Environment), specifically for your Python codes, is very important and much better than using notebooks.</blockquote><p>I also love to use Jupyter Notebook for Python. The interface is really convenient and the line-by-line coding really simplifies things and helps in quickly testing algorithms, but when you are writing a production-level code, the best practice is to use a good IDE. I prefer PyCharm. The reason is that it’s specifically built for Python and your scope gets a lot widened when you use it. It is easier to write a more modular, reusable code and make a proper project structure.</p><p>The most important advantage of using an IDE is that you can install plugins that actually improve coding. PyCharm already follows PEP 8 style and provides you with suggestions, when you are working on the code, for improving your code and keeping it up to standard. The other advantage, if you are using PyCharm, is that it saves local changes to your code on the basis of timestamp. This way, you can go to that particular timestamp to track your changes, and this becomes really useful if you can not remember which changes were done when. Here’s a GIF to show how local history can be accessed to check your changes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZGhAgpJ0TJHxzJjR9SCe_w.gif" /><figcaption><a href="https://blog.jetbrains.com/idea/2020/02/local-history-in-intellij-idea-may-save-your-life-code/">Using Local History to Restore Code Fragments or Deleted Files</a></figcaption></figure><p>Furthermore, PyCharm has a really good debugger. You can go through freecodecamp.org’s <a href="https://www.freecodecamp.org/news/debugging-with-pycharm-be282bcd6172/">tutorial</a> for learning how to use it.</p><h3>Group Files on the Basis of Tasks/Steps</h3><blockquote>A very useful way to keep the project code modular is by grouping files in terms of the overall tasks they perform.</blockquote><p>There can be a configuration file that has all the directories, paths, and values/strings that have to be hard-coded, and a helper file or a loading file that would have all the specific functions for loading files from those directories. There can be a commons.py for all the functions that are most called upon throughout the project. Then there will be your task-specific files that will call the functions from the common-function files.</p><p>You need to separate the reusable functions in a common file and import functions to your final code so that all the modules can access and use/reuse the functions efficiently.</p><h3>Use config.Py for All Your Directories</h3><blockquote>If your work requires you to load a lot of files or go to specific paths, you should mention all such directories or hard-coded values in a configuration file.</blockquote><p>This way, if there’s even a small change in one of the paths, you change it in only one place where you wrote it. Believe me, if you hard code paths at all instances you are loading files, just a small route change will turn out to be a troublesome task. In such a scenario, if there would be a change, you’d have to make the change everywhere. Fixing a directory, all those slashes, is not only tedious but can cause mistakes in one instance or another. So better to just save directories like below in a variable and later just load them for use:</p><pre>Config.py &gt;&gt; </pre><pre>MY_FILE_PATH = &quot;./abc/xyz.csv&quot;</pre><pre>Task_file &gt;&gt;</pre><pre>import config<br>pandas.read_csv(config.MY_FILE_PATH)</pre><h4>Make Config Files for All Environments</h4><p>When you are writing code later to be deployed in production, you need to test it many many times in the local environment as well as in the production environment. The root paths for production and local environments are not necessarily the same and you can not change the paths every time you need to test your code in either of the environments.</p><blockquote>You need to create separate config files for your local, test, and production environments and use a method to programmatically use a particular config file in terms of the environment you are in (like using cross-env in node.js).</blockquote><p>One of the methods you could use in Python for your environments is Argparse. The documentation for Argparse can be found <a href="https://docs.python.org/3/howto/argparse.html">here</a>.</p><h3>Use Blueprints for Flask</h3><p>In many of the python-based machine learning projects, the flask is used to create the server and build the model for the front-end to use it. If different routes and view functions need to be defined, it is better to use flask blueprints to structure your projects properly. You can go through a very useful guide <a href="https://realpython.com/flask-blueprint/">here</a>.</p><blockquote>Flask Blueprints are a great tool for dealing with application complexity as it increases.</blockquote><p>They can be used to ease the <strong>reusability</strong> of your own and third-party components and organize the overall application code much more efficiently.</p><h3>Try to Optimize Your Code As Much As Possible</h3><p>Code Optimization means you need to reduce the application’s processing time while using the least amount of resources possible. This can be done if faster methods are used to get stuff done and by avoiding redundancies. For example, in the above code of adding two lists element-wise, you could also have taken the route of using a for loop with zip() and appended the added numbers into the new list. The end result was the same but it was a slower and less efficient method which was simply avoided by using an already given function of NumPy i.e., <em>numpy.add(list1, list2).</em></p><p>You can always improve your code by using much more efficient functions or algorithms that would do the same tasks in a much shorter time while optimizing your system’s memory as well.</p><blockquote>Code Optimization at the production level becomes really important because the user or the client would demand a much-reduced application processing time on simpler non-dev machines.</blockquote><p>Sometimes slower coding methods are unavoidable especially in cases such as new processor architectures or limited/secure access production environments. Recently Apple shifted to an in-house ARM processor for their Computer Lineup. While the performance boost for normal users is incredible, developers face the challenge of incompatible libraries, because they have not yet been built for the new architecture. In such situations, although you have the libraries, you cannot use them. So you may have to go with alternate solutions which may slow down the overall performance. Therefore it’s always important to write the code and go for the most efficient methods, wherever possible, right from the get-go.</p><h3>Logging Critical Failures and Intermediate Results</h3><p>Logging is very important when writing production-level code. You need to track critical failures in your application and find a necessary solution to them. You can’t get the errors displayed using print statements in production-level code because you are not on the terminal anymore.</p><blockquote>I prefer to create a log text file that will show me any error tracebacks or intermediate results. The purpose for getting intermediate results logged is that you can gauge how the app is running and which part has been reached.</blockquote><p>For logging, one way is adding the following piece of code (in your main app.py file where you create the server and register blueprints):</p><pre>//define a path for your log file<br>logs_file_path = &quot;./app.log&quot;</pre><pre>//Create the log file<br>sys.stdout = open(logs_file_path, &quot;w&quot;, buffering=1, encoding=&#39;utf-8&#39;)</pre><p>This file will contain all the print statements in your application and will display the error tracebacks.</p><h3>Handle Possible Errors in the Code</h3><blockquote>Use Try and Except statements to handle possible errors and log them using <strong>traceback</strong>.</blockquote><p>Try and Except can be used in your overall flask view functions or in the most critical places you expect to find errors.</p><p>You can do all this as given below:</p><pre>import traceback<br>import sys</pre><pre>logs_file_path = &quot;./app.log&quot;<br>sys.stdout = open(logs_file_path, &quot;w&quot;, buffering=1, encoding=&#39;utf-8&#39;)</pre><pre>thisdict = {<br>    &quot;brand&quot;: &quot;Ford&quot;,<br>    &quot;model&quot;: &quot;Mustang&quot;,<br>    &quot;year&quot;: 1964<br>  }</pre><pre>try:  <br>  print(thisdict[&quot;car&quot;])</pre><pre>except Exception as i:<br>  print(traceback.format_exc())</pre><p>You can see that in the print statement, the wrong key has been used. The app.log file will display the traceback as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*9w2N_kbvfVG47r2CZY9BPQ.png" /></figure><h3>Proper Naming to Improve Readability</h3><p>The code you write should always have proper and understandable naming conventions followed for others to understand what’s happening. For example, if you are modifying a list and saving a new list, avoid using names like list2, list3, or new_list new_list1.</p><blockquote>The name should at most parts explain itself by its name.</blockquote><p>When we added the two lists, the name for the new list was <em>summed_list. </em>Anyone reading the code would get the idea that <em>summed_list</em> is a list made out of an addition operation. Proper naming should be used for function names as well. Try to keep a balance between going with a too-general name and a too-specific one.</p><h3>Create a Proper Project Structure</h3><blockquote>Making a great project structure is one of the most important things because that actually aligns your tasks in proper order, and you can make sure that modules are separated from each other properly.</blockquote><p>You need the source code folder where each module should have a separate directory with the main, say <em>module_1.py</em>, file in it that calls all the other functions in one place. The commons.py, config.py, and helper.py should be outside the module directory in your source code folder. This structure could look as below:</p><pre>Project_XYZ<br>|<br>|__app.py<br>|<br>|__templates<br>|<br>|__venv<br>|<br>|__configs<br>|     |<br>|     |__config.local.py<br>|     |<br>|     |__config.prod.py<br>|<br>|__src<br>      |<br>      |__module_1<br>      |          |__module_1.py<br>      |          |____init__.py<br>      |__module_2<br>      |          |__module_2.py<br>      |          |____init__.py<br>      |<br>      |__commons.py<br>      |<br>      |__input_loader.py<br>      |<br>      |__output_helper.py</pre><p>If you have anything to add or want to share some of your experience, please comment below.</p><h3>Previous Articles</h3><ul><li><a href="https://medium.com/red-buffer/integrating-python-flask-backend-with-electron-nodejs-frontend-8ac621d13f72">Integrating Python Flask Backend with Electron (Nodejs) Frontend</a></li><li><a href="https://medium.com/red-buffer/electron-builder-packaging-electron-nodejs-application-along-with-flask-app-for-windows-fc26f5a29870">Electron Builder: Packaging Electron (Nodejs) Application (along with Flask app) for Windows</a></li></ul><h3>About Ahmad Sachal</h3><ul><li>LinkedIn Profile — <a href="https://www.linkedin.com/in/ahmad-sachal/">linkedin.com/in/ahmad-sachal/</a></li></ul><h3>About Red Buffer</h3><ul><li>Website — <a href="http://redbuffer.net/">redbuffer.net</a></li><li>LinkedIn Page — <a href="https://www.linkedin.com/company/red-buffer">linkedin.com/company/red-buffer</a></li></ul><h3>References for Learning</h3><ul><li><a href="https://realpython.com/flask-blueprint/">Use a Flask Blueprint to Architect Your Applications</a></li><li><a href="https://docs.quantifiedcode.com/python-anti-patterns/index.html">The Little Book of Python Anti-Patterns</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c39246e0233" width="1" height="1" alt=""><hr><p><a href="https://medium.com/red-buffer/python-production-level-coding-practices-4c39246e0233">Python: Production-Level Coding Practices</a> was originally published in <a href="https://medium.com/red-buffer">Red Buffer</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Electron Builder: Packaging Electron (Nodejs) Application (along with Flask app) for Windows]]></title>
            <link>https://medium.com/red-buffer/electron-builder-packaging-electron-nodejs-application-along-with-flask-app-for-windows-fc26f5a29870?source=rss-85ddcd6488de------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc26f5a29870</guid>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[windows-app-development]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[electron]]></category>
            <dc:creator><![CDATA[Ahmad Sachal]]></dc:creator>
            <pubDate>Fri, 28 May 2021 12:18:06 GMT</pubDate>
            <atom:updated>2021-05-28T12:18:06.636Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/735/0*9F-zfFsiM46zmqpe.jpg" /></figure><p>In the previous article, we discussed how to integrate an electron app frontend with a Flask backend but it did not cover how to package the Electron app. So this piece elucidates how an Electron app can be packaged for Windows using <a href="https://www.electron.build/"><strong>Electron Builder</strong></a>. There are a few other libraries that are used to package the Electron app as well, like electron-packager, but electron builder is a more preferred choice here for building the Electron app because it has a wide range of options, which is very useful for a complex application.</p><p>If you are here, there’s a high chance you already have your project ready to be packaged. If this isn’t the case, you need to first create the Electron app. The basic guide to creating a simple quickstart Electron application is given in the Electronjs <a href="https://www.electronjs.org/docs/tutorial/quick-start">docs</a>. You can also go to my previous <a href="https://medium.com/red-buffer/integrating-python-flask-backend-with-electron-nodejs-frontend-8ac621d13f72">article</a> which covers the pre-requisites for this article.</p><p>We know that the project structure for a basic Electron app is as:</p><pre>my-electron-app/ <br>├── package.json <br>├── main.js <br>├── renderer.js <br>└── index.html</pre><p>For building an installable electron app with electron builder, your main concern is with <em>package.json</em>.</p><h4>Installing Electron Builder</h4><p>To install electron builder, the docs recommend using yarn instead of npm. The package can be installed using the following command:</p><pre>yarn add electron-builder — dev</pre><h3>Adding Build Configurations in package.json</h3><p>The electron builder docs cover all aspects and features, so I’ll be more specific to our use case.</p><p>You need to specify the standard fields for the app like name, description, product name, version, author in package.json. This gets covered when the project is initiated with package.json.</p><h4>App ID, Directories, and Specifications for Windows</h4><p>This is the most important step and needs to be implemented properly. In package.json, you need to specify the app ID, the output and build resource directories and, most importantly, the OS you are going for. In our case, it’s Windows, and let’s say we want the <em>NSIS</em> file as our build file. We will add the following in the file:</p><pre>&quot;build&quot;: {<br>    &quot;appId&quot;: &quot;electron-test-app&quot;,    <br>    &quot;directories&quot;: {<br>                    &quot;buildResources&quot;: &quot;resources&quot;,<br>                    &quot;output&quot;: &quot;release&quot;<br>                   },<br>    &quot;win&quot;: {<br>            &quot;publisherName&quot;: &quot;xyz.net&quot;,<br>            &quot;target&quot;: [<br>                       {<br>                         &quot;target&quot;: &quot;nsis&quot;,<br>                         &quot;arch&quot;: [<br>                           &quot;x64&quot;<br>                          ]<br>                       }<br>                      ]</pre><pre>           },</pre><p>You can change these specifications according to your requirements by going through the docs. You can go for an <em>MSI</em> file as well or go for both.</p><h4>Specifying Files and Resources</h4><p>The build needs to be told which files or source codes will be required by the app to run smoothly. All the required resources need to be specified within the <em>build </em>as:</p><pre>&quot;build&quot;: {     <br>     ...</pre><pre>     &quot;files&quot;: [<br>           &quot;./resources/icon.*&quot;,<br>           &quot;./src/&quot;,<br>           &quot;./css/&quot;,<br>           &quot;./contents/&quot;,<br>           &quot;./node_modules/**/*&quot;,<br>           &quot;./package.json&quot;<br>      ],</pre><h4>For Static Files That Need to be Directly Used from the App Directory</h4><p>Some of the resource files are required to be copied as app resources for usage. It could be an excel sheet for instance. Electron builder has great options for this. You can use:</p><ul><li><strong>extraResources:</strong> copies the files (keeping their exact name) to the app’s resource directory</li><li><strong>extraFiles:</strong> copies the files directly into the app’s content directory</li></ul><p>The files can be specified in the build as shown below:</p><pre>&quot;build&quot;: {     <br>     ...</pre><pre>     &quot;extraFiles&quot;: [<br>      {<br>        &quot;from&quot;: &quot;./cache/&quot;,<br>        &quot;to&quot;: &quot;cache/&quot;,<br>        &quot;filter&quot;: [<br>          &quot;**/*&quot;<br>         ]<br>      },</pre><p>extraResources is also very similar to extraFiles. It’s just a matter of what the requirement is for the app.</p><h4><em>Notes Worth Mentioning Regarding Directories</em></h4><p>While using electron builder, I encountered some issues that I feel needed to be mentioned. There might be better solutions but I’ll specify what I did or preferred particularly.</p><ul><li>Empty directories will not get copied to the required locations. You could add a keep.txt file in the directory to copy the folders without any issues.</li><li>You might want to avoid creating folders programmatically from the app since that causes errors and problems. The best thing in my experience was to just copy such folders to the required place using extraFiles or extraResources.</li></ul><h4>Running the Application After Installation</h4><p>It’s the developers choice whether to run the application right after installation or not. In my experience, the application has a lot of components and libraries installed and will have some issues when it will run the first time, but no issues when it runs again. So I went on to avoid running the application right after installation. For setting this option, you need to add the following line in the <em>build-in</em> package.json:</p><pre>&quot;build&quot;: {     <br>   ...</pre><pre>   &quot;nsis&quot;: {<br>      &quot;runAfterFinish&quot;: false<br>   },</pre><h3>Packaging the App Alongwith the Flask App</h3><p>The flask app use case is in relation to my previous article about <a href="https://medium.com/red-buffer/integrating-python-flask-backend-with-electron-nodejs-frontend-8ac621d13f72">Integrating Python Flask Backend with Electron (Nodejs) Frontend</a>. This part will cover packaging another executable app with your main application. We have a simple flask executable for this. You can just skip the next step if you are not working with another executable app within your main app.</p><h4>Specifying Flask App Resource</h4><p>We need to set the resource directory for our flask app right where the app would access it from its route directory. This is just the same step we discussed before. We need to use extraFiles or extraResources for our task here. It can be done as below:</p><pre>&quot;extraFiles&quot;: [<br>      {<br>        &quot;from&quot;: &quot;./backend/dist/app&quot;,<br>        &quot;to&quot;: &quot;backend/dist/app/&quot;,<br>        &quot;filter&quot;: [<br>          &quot;**/*&quot;<br>         ]<br>      },</pre><p>Flask app created by Pyinstaller is in the<em> dist</em> folder and <em>backend</em> is just the directory of the backend python code.</p><p>The important thing is to maintain the project structure for the application by specifying the resources in their exact locations w.r.t. the app’s route directory. Make sure the path to the new directory, copied with extraFiles or extraResources, is being correctly accessed in the app code, to avoid issues.</p><h4>Creating the Build</h4><p>You can either directly run the command from your terminal or set a specific key for your command in the script. The command to build the app for Windows 64-bit is:</p><pre>electron-builder build --win --x64</pre><p>You can add the following in <em>scripts </em>to just set the key for this command and run it when you want to package the app:</p><pre>&quot;scripts&quot;: {<br>      &quot;package&quot;: &quot;electron-builder build --win --x64&quot;<br>},</pre><p>Run <em>yarn package </em>in your terminal and the app will get built. For just the unpacked version, run <em>yarn package --dir.</em></p><p>There are many options to go for with electron builder for different use cases. The package documentation can be consulted for that. Electron builder also has <strong>Auto Update </strong>options that can be explored as well so that the user keeps getting the updated version.</p><h3>Previous Article</h3><ul><li><a href="https://medium.com/red-buffer/integrating-python-flask-backend-with-electron-nodejs-frontend-8ac621d13f72">Integrating Python Flask Backend with Electron (Nodejs) Frontend</a></li></ul><h3>About Ahmad Sachal</h3><ul><li>LinkedIn Profile — <a href="https://www.linkedin.com/in/ahmad-sachal/">linkedin.com/in/ahmad-sachal/</a></li></ul><h3>About Red Buffer</h3><ul><li>Website — <a href="http://redbuffer.net/">redbuffer.net</a></li><li>LinkedIn Page — <a href="https://www.linkedin.com/company/red-buffer">linkedin.com/company/red-buffer</a></li></ul><h3>References for Learning</h3><ul><li><a href="https://github.com/electron/electron-quick-start">electron/electron-quick-start</a> — a very basic starter Electron app</li><li><a href="https://www.electron.build/">electron.build</a> — Electron Builder documentation</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc26f5a29870" width="1" height="1" alt=""><hr><p><a href="https://medium.com/red-buffer/electron-builder-packaging-electron-nodejs-application-along-with-flask-app-for-windows-fc26f5a29870">Electron Builder: Packaging Electron (Nodejs) Application (along with Flask app) for Windows</a> was originally published in <a href="https://medium.com/red-buffer">Red Buffer</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integrating Python Flask Backend with Electron (Nodejs) Frontend]]></title>
            <link>https://medium.com/red-buffer/integrating-python-flask-backend-with-electron-nodejs-frontend-8ac621d13f72?source=rss-85ddcd6488de------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ac621d13f72</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[electronjs]]></category>
            <category><![CDATA[javascript-development]]></category>
            <category><![CDATA[flask-framework]]></category>
            <category><![CDATA[integration]]></category>
            <dc:creator><![CDATA[Ahmad Sachal]]></dc:creator>
            <pubDate>Fri, 12 Mar 2021 07:32:22 GMT</pubDate>
            <atom:updated>2021-03-12T07:32:22.171Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*_qsPTTOGXlodELY_LXp-kQ.jpeg" /></figure><p>This article is aimed to be a consolidated guide on Flask and Electron integration. The goal is to cover all the important points for this task since the information for this use-case isn’t abundant nor is it all available in one place. Coming from someone who had to extensively research to solve the same problem very recently, I hope this article will be helpful.</p><p>The intention here is to stay specific to the problem of <em>integration</em>, so I am going to stay with a simple ‘Hello World’ program on the backend.</p><h4>Creating a Basic Electron App</h4><p>For creating a basic Electron quick-start app, you can go to <a href="https://www.electronjs.org/docs/tutorial/quick-start">electron JS doc</a> for guidance.</p><p>For now, let’s focus on the main.js code given in the docs that creates the app browser window. Here’s the piece of code from the link above that we are going to refer to for our article:</p><pre>const { app, BrowserWindow } = require(‘electron’)</pre><pre>function createWindow () {<br> const win = new BrowserWindow({<br> width: 800,<br> height: 600,<br> webPreferences: {<br> nodeIntegration: true<br> }<br> })</pre><pre>win.loadFile(‘index.html’)<br>}</pre><pre>app.whenReady().then(createWindow)</pre><pre>app.on(‘window-all-closed’, () =&gt; {<br> if (process.platform !== ‘darwin’) {<br> app.quit()<br> }<br>})</pre><pre>app.on(‘activate’, () =&gt; {<br> if (BrowserWindow.getAllWindows().length === 0) {<br> createWindow()<br> }<br>})</pre><p>The code creates an application window for our Electron app. This is where we are to integrate our Flask application with Electron.</p><h4>Creating a Basic Flask App</h4><p>Here is a simple Flask app code we are going to use for the backend application.</p><pre>from flask import Flask<br>app = Flask(__name__)<br><br><br>@app.route(&#39;/test&#39;)<br>def hello():<br>    return &quot;Hello World!&quot;<br><br>if __name__ == &#39;__main__&#39;:<br>    app.run()</pre><p>The default port would be 127.0.0.1:80, but you can change the port by modifying the last line as:</p><pre>app.run(host=’127.0.0.1&#39;, port=5000)</pre><p>Now for our problem, we are going to use the Child process. For those who already know about the Child process, skip the definition below.</p><p><strong><em>Child Process<br></em></strong><em>A </em><a href="https://www.techopedia.com/definition/26327/child-process"><em>Child process</em></a><em> is the creation of a Parent process, which can be defined as the main process that creates a Child or subprocesses to perform certain operations. Each process can have many Child processes but only one Parent. A child process inherits most of its Parent’s attributes.</em></p><p><em>There are multiple ways to create a Child process but for our case, we are going to stick with the following two methods:</em></p><ul><li><a href="https://nodejs.org/api/child_process.html#child_process_child_process_spawn_command_args_options"><em>spawn</em></a></li><li><a href="https://nodejs.org/api/child_process.html#child_process_child_process_execfile_file_args_options_callback"><em>execFile</em></a></li></ul><h4>spawn: Running Flask App As a Source Code (app.py)</h4><p>In case the developer wants to test the electron/Flask app without creating the Flask executable, they can use the spawn method. This can be done by writing the following code inside the create window function:</p><pre>var python = require(&#39;child_process&#39;).spawn(&#39;py&#39;, [&#39;./backend/app.py&#39;]);<br>  python.stdout.on(&#39;data&#39;, function (data) {<br>    console.log(&quot;data: &quot;, data.toString(&#39;utf8&#39;));<br>  });<br>  python.stderr.on(&#39;data&#39;, (data) =&gt; {<br>    console.log(`stderr: ${data}`); // when error<br>  });</pre><p>This is only for temporary use until the backend app is finalized. When the backend code is ready in accordance with the requirements, you need to make an executable file for the Flask app which the Electron application starts in its main process every time the Electron app is started.</p><h4>execFile: Running Flask App Executable in Electron (app.exe)</h4><p>To create the Flask app executable, you need pyinstaller. This can simply be installed by <strong><em>pip install pyinstaller</em></strong>. After installing pyinstaller, you need to go to the command line and type <strong><em>pyinstaller -F app.py.</em></strong></p><p>Note the <strong><em>-F </em></strong>is used here to build the app in only one file. You can choose not to do this. The executable app will be in the ‘dist’ folder in the backend code root directory.</p><pre>let backend;<br>backend = path.join(process.cwd(), &#39;resources/backend/dist/app.exe&#39;)</pre><pre>var execfile = require(‘child_process’).execFile;</pre><pre>execfile(<br> backend,<br> {<br>  windowsHide: true,<br> },</pre><pre> (err, stdout, stderr) =&gt; {</pre><pre>  if (err) {<br>  console.log(err);<br>  }</pre><pre>  if (stdout) {<br>  console.log(stdout);<br>  }</pre><pre>  if (stderr) {<br>  console.log(stderr);<br>  }<br> }<br>)</pre><h4>Killing app.exe When Electron App Quits</h4><p>You have successfully integrated the Flask app with Electron, you start the application, run your task and close the application, but oh wait… the Flask app is still running in the background. It never closed. This problem occurred when I had thought everything was now working well. There is not much specific information available for this purpose. I tried npm’s <em>tree-kill</em> for killing the Child process, but it didn’t work for me. The best solution I found was to create another Child process that is given the command to kill the running app. Following is the piece of code you need to put inside the <em>app.on(‘window-all-closed’) </em>function<em> </em>before the <em>app.quit() </em>line:</p><pre>const { exec } = require(‘child_process’);</pre><pre>exec(‘taskkill /f /t /im app.exe’, (err, stdout, stderr) =&gt; {</pre><pre> if (err) {</pre><pre>  console.log(err)</pre><pre> return;</pre><pre> }</pre><pre> console.log(`stdout: ${stdout}`);</pre><pre> console.log(`stderr: ${stderr}`);</pre><pre>});</pre><p>The code above worked well for me since it killed all the background processes when I quit my app.</p><h4>Communicating Between Backend and Frontend</h4><p>The best way to communicate between frontend and backend is to send requests to the Flask server endpoint from your UI with the click of a button or a user input. For this, you can use one of the more frequently used http methods. I usually prefer Axios for this. You need to use the following code in your Renderer.js while sending requests to the Flask server.</p><pre>test = &#39;Hello&#39;<br>async function makePostRequest(test) {</pre><pre> axios.post(‘http://127.0.0.1:5000/test&#39;, test)</pre><pre> .then(function (response) {</pre><pre>  console.log(“It says: “, response.data);</pre><pre> })</pre><pre> .catch(function (error) {</pre><pre>  console.log(error);</pre><pre> });</pre><pre>}</pre><p>You will find the response from the Flask server in your app log. You can perform further operations inside the <em>.then() </em>code.</p><p>When the frontend is ready, you can build the installable electron application efficiently by using an <a href="https://www.electron.build/">electron-builder</a>. Go to the docs in the given link, make the necessary changes in your package.json file and run the build command in your terminal to build your application.</p><h4>Avoid Route Directory Mistake</h4><p>Note that since the Flask app executable is going to be opened in the Electron app environment, the paths inside the flask app code need to be referenced in accordance with the Electron app root directory. The mistake of mentioning file paths relative to the Flask environment, which will be alien to the Electron environment, can be made. So if your Flask app loads a file, you need to specify the path according to Electron and not according to your Flask environment.</p><h3>About Ahmad Sachal</h3><ul><li>LinkedIn Profile—<a href="https://www.linkedin.com/in/ahmad-sachal/">linkedin.com/in/ahmad-sachal/</a></li></ul><h3>About Red Buffer</h3><ul><li>Website — <a href="http://redbuffer.net/">redbuffer.net</a></li><li>Linkedin Page — <a href="https://www.linkedin.com/company/red-buffer">linkedin.com/company/red-buffer</a></li></ul><h3>References for Learning</h3><ul><li><a href="https://realpython.com/flask-blueprint/">realpython.com/flask-blueprint/</a> — Use a Flask Blueprint to Architect Your Applications</li><li><a href="https://github.com/electron/electron-quick-start">electron/electron-quick-start</a> — a very basic starter Electron app</li><li><a href="https://nodejs.org/api/child_process.html">nodejs.org/api/child_process.html</a> — Child Process documentation</li><li><a href="https://www.npmjs.com/package/axios#example">www.npmjs.com/package/axios#example</a> — Make Get/Post requests with Axios</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ac621d13f72" width="1" height="1" alt=""><hr><p><a href="https://medium.com/red-buffer/integrating-python-flask-backend-with-electron-nodejs-frontend-8ac621d13f72">Integrating Python Flask Backend with Electron (Nodejs) Frontend</a> was originally published in <a href="https://medium.com/red-buffer">Red Buffer</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>