<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Barış Özmen on Medium]]></title>
        <description><![CDATA[Stories by Barış Özmen on Medium]]></description>
        <link>https://medium.com/@barisozmen?source=rss-4f549af76ae------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*08NSpngLytl1h224ShGRsA.jpeg</url>
            <title>Stories by Barış Özmen on Medium</title>
            <link>https://medium.com/@barisozmen?source=rss-4f549af76ae------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 01:41:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@barisozmen/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Laws of Programming]]></title>
            <link>https://medium.com/@barisozmen/laws-of-programming-cbec3559f12f?source=rss-4f549af76ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbec3559f12f</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Barış Özmen]]></dc:creator>
            <pubDate>Sat, 04 Jan 2025 17:31:39 GMT</pubDate>
            <atom:updated>2025-01-04T18:18:51.028Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/495/0*ggc8HNg9xGII5EQb.png" /><figcaption><a href="https://xkcd.com/974/">xkcd.com/974</a></figcaption></figure><p>Programming laws are distilled wisdom from hard-earned lessons from the trenches. They provide a powerful shorthand for hackers to avoid the same mistakes that have been made time and time again. I’ve grouped them by topic below, and prioritized by how much they’ve saved me from wasted effort.</p><h3>Avoid early optimization</h3><ul><li><a href="https://www.youtube.com/watch?v=74RdET79q40">Knuth’s Optimization Principle</a>: Premature optimization is the root of all evil.</li><li><a href="https://www.martinfowler.com/bliki/BeckDesignRules.html">Kent Beck</a> — “Make it run, then make it right, then make it fast.”</li><li><a href="https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it">YAGNI (You Aren’t Gonna Need It):</a> Don’t implement functionality until it’s necessary.</li></ul><h3>Iterate fast, launch early</h3><ul><li><a href="https://blog.codinghorror.com/boyds-law-of-iteration/">Boyd’s Law of iteration</a>: speed of iteration beats quality of iteration.</li><li><a href="https://yiming.dev/clipping/2019/03/21/le-blanc&#39;s-law-a-k-a-later-equals-never/">LeBlanc’s Law</a>: later equals never</li><li><a href="https://en.wikipedia.org/wiki/Unix_philosophy">Third Rule of Unix Philosophy</a>: Design and build software, even operating systems, to be tried early, ideally within weeks. Don’t hesitate to throw away the clumsy parts and rebuild them.</li><li><a href="http://principles-wiki.net/principles:gall_s_law">Gall’s Law</a>: A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system.</li></ul><h3>Done is better than perfect</h3><ul><li><a href="https://www.reddit.com/r/productivity/comments/q2o1ud/done_is_better_than_perfect/">Done is better than perfect</a>: for morale, and also for avoiding early optimization.</li><li><a href="https://en.wikipedia.org/wiki/Worse_is_better">Worse is Better:</a> Simplicity of implementation is more important than correctness or completeness.</li></ul><h3>Keep it stupidly simple</h3><ul><li><a href="https://www.laws-of-software.com/laws/kernighan/">Kernighan’s Law</a>: “Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.”</li><li><a href="http://www.robelle.com/smugbook/laws.html#:~:text=Hoare&#39;s%20Law%20of%20Large%20Programs,program%20struggling%20to%20get%20out.">Hoare’s Law of Large Programs</a>: inside every large program is a small program struggling to get out.</li><li><a href="https://en.wikipedia.org/wiki/KISS_principle">KISS (Keep It Simple, Stupid)</a>: Simple solutions are better; avoid unnecessary complexity.</li><li><a href="https://en.wikipedia.org/wiki/Albert_Einstein">Einstein</a> — <em>“Everything should be as simple as possible but not simpler.”</em></li><li><a href="https://en.wikipedia.org/wiki/Antoine_de_Saint-Exup%C3%A9ry">Antoine de Saint-Exupéry</a> — <em>“Perfection (in design) is achieved not when there is nothing more to add, but rather when there is nothing more to take away.“</em></li></ul><h3>Have an eyesight on complexity</h3><ul><li><a href="https://en.wikipedia.org/wiki/Lehman%27s_laws_of_software_evolution#:~:text=(1974)%20%22Increasing%20Complexity%22,to%20maintain%20or%20reduce%20it.">Lehman’s Law of increasing complexity</a>: Complexity grows unless work is done to reduce it.</li><li><a href="https://en.wikipedia.org/wiki/Second-system_effect">Second-System Effect</a>: tendency of small, elegant, and successful systems to be succeeded by over-engineered, bloated systems, due to inflated expectations and overconfidence</li><li><a href="https://en.wikipedia.org/wiki/Jamie_Zawinski#:~:text=Zawinski&#39;s%20Law%20of%20Software%20Envelopment,replaced%20by%20ones%20which%20can.">Zawinski’s Law</a>: Every program attempts to expand until it can read mail.</li><li><a href="https://en.wikipedia.org/wiki/Scope_creep">Scope Creep</a>: The gradual expansion of a project’s goals and tasks beyond its original objectives</li><li><a href="https://en.wikipedia.org/wiki/Brian_Kernighan">Brian Kernighan</a> — <em>“Controlling complexity is the essence of computer programming”</em></li></ul><h3>Expect it to take longer than expected</h3><ul><li><a href="https://en.wikipedia.org/wiki/Ninety%E2%80%93ninety_rule">The 90–90 Rule</a>: The first 90% of code accounts for 90% of the development time; the last 10% takes another 90% of the time.</li><li><a href="https://en.wikipedia.org/wiki/Hofstadter%27s_law">Hofstadter’s Law</a>: It always takes longer than you expect, even when you account for Hofstadter’s Law.</li></ul><h3>Design well</h3><h4>Think your code as lego pieces — Composability</h4><ul><li><a href="https://en.wikipedia.org/wiki/Unix_philosophy">First Rule of Unix Philosophy</a>: Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new “features”.</li><li><a href="https://en.wikipedia.org/wiki/Single-responsibility_principle">Single Responsibility Principle</a>: A class or module should have one, and only one, reason to exist.</li><li><a href="https://en.wikipedia.org/wiki/Robustness_principle">Postel’s law (robustness principle)</a> : “Be liberal in what you accept, and conservative in what you send.”</li><li><a href="https://en.wikipedia.org/wiki/Separation_of_concerns">Separation of Concerns</a>: Different aspects of a program should be managed by separate modules or functions.</li><li><a href="http://principles-wiki.net/principles:principle_of_separate_understandability">Separate Understandability Principle</a>: Each module shall be understandable on its own — without knowing anything about other modules.</li><li><a href="https://en.wikipedia.org/wiki/Loose_coupling">Loose Coupling</a>: Each component can function independently and make changes without significantly affecting the others</li><li><a href="https://en.wikipedia.org/wiki/Law_of_Demeter">Law of Demeter (Principle of Least Knowledge)</a>: a specific case of loose coupling.</li><li><a href="https://cdixon.mirror.xyz/Ie8BQeiM1JTJkKISPVaAao88-XVMoHsdUAtcnS9_Jbs">Composability enables compounding</a>: <em>“Composability is to software as compounding interest is to finance”</em> –<a href="https://cdixon.mirror.xyz/">cdixon</a></li></ul><h4>Be intuitive</h4><ul><li><a href="https://en.wikipedia.org/wiki/Don%27t_Make_Me_Think">Don’t Make Me Think</a>: Designs should be self-evident and intuitive.</li><li><a href="https://en.wikipedia.org/wiki/Principle_of_least_astonishment#:~:text=In%20user%20interface%20design%20and,not%20astonish%20or%20surprise%20users.">Principle of Least Astonishment</a>: Systems should behave in a way that least surprises the user.</li><li><a href="https://en.wikipedia.org/wiki/Self-documenting_code#:~:text=Self%2Ddocumenting%20code%20is%20ostensibly,symbol&#39;s%20meaning%2C%20such%20as%20article.">Self-documenting code</a>: name your variables and functions for ease of understanding of people reading your code in the future (including yourself)</li></ul><h4>Data model &gt; algorithms</h4><ul><li><a href="https://en.wikipedia.org/wiki/Linus_Torvalds">Linus Torvalds</a>: <em>“Bad programmers worry about the code. Good programmers worry about data structures and their relationships.”</em></li><li><a href="https://en.wikipedia.org/wiki/Rob_Pike">Rob Pikes</a>: <em>“Data dominates. If you’ve chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming.”</em></li><li><a href="http://www.catb.org/esr/">Eric Raymond</a> (in <a href="https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar">The Cathedral and the Bazaar</a>) — <em>“Smart data structures and dumb code work a lot better than the other way around.“</em></li></ul><h3>Debugging</h3><ul><li><a href="https://en.wikipedia.org/wiki/Fail-fast_system#:~:text=Fail%2Dfast%20architectures%20are%20based,in%20a%20fault%2Dtolerant%20system.">Fail Fast Principle:</a> Crash early when something goes wrong. It’s better to face problems head-on than let them fester.</li><li><a href="https://en.wikipedia.org/wiki/Linus%27s_law#:~:text=In%20software%20development%2C%20Linus&#39;s%20law,in%20honor%20of%20Linus%20Torvalds.">Linus’s Law</a>: Given enough eyeballs, all bugs are shallow</li><li><a href="https://en.wikipedia.org/wiki/Murphy%27s_law">Murphy’s Law</a>: Anything that can go wrong will go wrong. Write your code to expect failure, not sunshine and rainbows.</li></ul><h3>Profiling: Look for the bottlenecks</h3><ul><li><a href="https://en.wikipedia.org/wiki/Amdahl%27s_law">Amdahl’s Law:</a> the overall performance improvement gained by optimizing a single part of a system is limited by the fraction of time that the improved part is actually used</li><li><a href="https://en.wikipedia.org/wiki/Liebig%27s_law_of_the_minimum">Liebig’s Law of the Minimum</a>: growth is dictated not by total resources available, but by the scarcest resource</li></ul><h3>Trust the old</h3><ul><li><a href="https://en.wikipedia.org/wiki/Lindy_effect">Lindy Effect</a>: Code that’s been around for a while is likely stable. Don’t mess with it unless you have a good reason.</li><li><a href="https://en.wikipedia.org/wiki/G._K._Chesterton">Chesterton’s Fence</a>: Before you remove or refactor code, make sure you understand why it was there in the first place. Deleting unknowns invites new bugs.</li></ul><p>.</p><p>[This article is mirrored at <a href="https://bozmen.io/laws">bozmen.io</a>, <a href="https://mirror.xyz/0xA9C3cc3BeacF8Dd77DDB9349523892FA0f026403/kKVFPbxntm_-rfcgTsMPMkKwg_Jt1xAsoTlGoXYxLcs">mirror.xyz</a> and <a href="https://bozmen.substack.com/">bozmen.substack.com</a>]</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbec3559f12f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AutoML for Data Augmentation]]></title>
            <link>https://medium.com/insight-data/automl-for-data-augmentation-e87cf692c366?source=rss-4f549af76ae------2</link>
            <guid isPermaLink="false">https://medium.com/p/e87cf692c366</guid>
            <category><![CDATA[automl]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[convolution-neural-net]]></category>
            <category><![CDATA[insight-ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Barış Özmen]]></dc:creator>
            <pubDate>Wed, 27 Mar 2019 15:25:37 GMT</pubDate>
            <atom:updated>2019-05-03T21:16:06.639Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UtE8TlrOrZDNlE5lfKwfsw.png" /></figure><p><a href="https://github.com/barisozmen/deepaugment">DeepAugment</a> is an AutoML tool focusing on data augmentation. It utilizes Bayesian optimization for discovering data augmentation strategies tailored to your image dataset. The main benefits and features of DeepAugment are:</p><ul><li><strong>Reduces the error rate of CNN models</strong> (showed 60% decrease in error for CIFAR10 on WRN-28–10)</li><li><strong>Saves time</strong> by automating the process</li><li><strong>50 times faster</strong> than Google’s previous solution–<a href="https://ai.googleblog.com/2018/06/improving-deep-learning-performance.html">AutoAugment</a></li></ul><p>The finished package is on <a href="https://pypi.org/project/deepaugment/">PyPI</a>. You can install it from your terminal by running:</p><pre>$ pip install deepaugment</pre><p>You can also visit the project’s <a href="https://github.com/barisozmen/deepaugment">README</a> or run the Google Colab notebook <a href="http://bit.ly/deepaugmentusage">tutorial</a>. To learn more about how I built this, read on!</p><h3>Introduction</h3><p>Data is the most critical piece of AI applications. Not having enough labeled data often leads to overfitting, which means the model will not be able to generalize to unseen examples. This can be mitigated by data augmentation, which effectively increases the amount and diversity of data seen by the network. It is done by artificially producing new data by applying transformations on an original dataset such as rotation, cropping, occlusion, etc. However, determining which augmentations will work best for the dataset at hand is no trivial task. To address this problem, Google published <a href="https://ai.googleblog.com/2018/06/improving-deep-learning-performance.html">AutoAugment</a> last year, which discovers optimized augmentations for the given dataset using <a href="https://en.wikipedia.org/wiki/Reinforcement_learning">reinforcement learning</a>.</p><p>Using Google’s AutoAugment requires powerful computational resources due to the reinforcement learning module. Since obtaining the necessary computational power can be costly, I developed a novel approach, <a href="https://github.com/barisozmen/deepaugment">DeepAugment</a>, which employs <a href="https://app.sigopt.com/static/pdf/SigOpt_Bayesian_Optimization_Primer.pdf">Bayesian optimization</a> instead of reinforcement learning.</p><h3>Ways to get better data</h3><p>Efforts to improve the quality of data often have a higher return on investment than efforts to enhance models. There are three main ways to improve data: <strong>collecting</strong> more data, <strong>synthesizing</strong> new data, or <strong>augmenting</strong> existing data. Collecting additional data is not always possible and can be expensive. Data synthesis, done by <a href="https://arxiv.org/abs/1406.2661">GANs</a>, is promising but complicated, and might <a href="https://www.intel.ai/fake-it-till-you-make-it-synthetic-datasets-assisting-machine-learning-in-data-scarce-environments/">diverge from realistic examples</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_yBsFnC3Jq9McRLUI684zg.png" /></figure><p>Data augmentation, on the other hand, is simple and has high impact. It is applicable to most datasets and is done with simple image transformations. The problem, however, is determining which augmentation technique is best for the dataset at hand. Discovering the proper method requires time-consuming experimentation. Even after many experiments, a machine learning (ML) engineer may still not discover the best option.</p><p>Effective augmentation strategies are different for each image dataset, and some augmentation techniques may even be detrimental to the model. For example, applying rotations would make your model worse if you are using it with the <a href="http://yann.lecun.com/exdb/mnist/">MNIST digits dataset</a>, because a 180 degree rotation on a “6” would make it look like a “9”, while still being labeled as a 6. On the other hand, applying rotation to satellite images can improve results significantly since a car image from the air will still be a car, no matter how much it is rotated.</p><h3>DeepAugment: lightning fast autoML</h3><p>DeepAugment is designed as a fast and flexible autoML data augmentation solution. More specifically, it is designed as a faster and more flexible alternative to AutoAugment (<a href="https://arxiv.org/abs/1805.09501">Cubuk et al., 2018</a>, <a href="https://ai.googleblog.com/2018/06/improving-deep-learning-performance.html">blog</a>). AutoAugment was one of the most exciting publications in 2018, and the first method using reinforcement learning for this particular problem. At the time of this article, <a href="https://github.com/tensorflow/models/tree/master/research/autoaugment">the open source version of AutoAugment</a> did not provide the controller module, which prevents users from running it for their own datasets. Moreover, it takes <strong>15,000</strong> iterations to learn augmentation policies, requiring huge computational resources. Most people could not benefit from it even if its source code was fully available.</p><p>DeepAugment addresses these problems with the following design goals:</p><ol><li><strong>Minimize the computational complexity</strong> of the optimization of data augmentation while maintaining the quality of results.</li><li>Be <strong>modular and user-friendly.</strong></li></ol><p>In order to achieve the first goal, DeepAugment was designed with the following differences, as compared to AutoAugment:</p><ul><li>Utilizes Bayesian optimization instead of reinforcement learning (requires fewer iterations) (~100x speed-up)</li><li>Minimizes size of child model (decreases computational complexity of each training) (~20x speed-up)</li><li>Less stochastic augmentation search space design (decreases number of iterations needed)</li></ul><p>To achieve the second goal, making DeepAugment modular and user-friendly, the user interface is designed in a way that gives the user a broad configuration of possibilities and model selections (e.g. selecting the child model or inputting a self-designed child model, see <a href="https://github.com/barisozmen/deepaugment#configuration-options">configuration options</a>).</p><h4>Designing augmentation policies</h4><p>DeepAugment aims to find the best augmentation policy for a given image dataset. An augmentation policy is defined as the sum of five sub-policies, which are made from two types of augmentation techniques and two real-values [0, 1], determining how powerfully each augmentation technique will be applied. I implemented augmentation techniques using the <a href="https://github.com/aleju/imgaug">imgaug</a> package, which is known for its large collection of augmentation techniques (see below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kqKzGryy0qYjEXbR3kyNOg.png" /></figure><p>Augmentations are most effective when they are diverse and randomly applied. For instance, instead of rotating every image, it is better to rotate some portion of images, shear another portion, and apply a color inversion for another. Based on this observation, DeepAugment applies one of five sub-policies (consisting of two augmentations) randomly to the images. During the optimization process, each image has an equal chance (16%) of being augmented by one of five sub-policies and a 20% chance of not being augmented at all.</p><p>While I was inspired by AutoAugment for this policy design, there is one main difference: I do not use any parameters for the probability of applying sub-policies in order to make policies less stochastic and allow optimization in fewer iterations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pcvN13iGLZl6k2AOyRYgJQ.png" /></figure><p>This policy design creates a 20-dimensional search space for the Bayesian optimizer, where 10 dimensions are categorical (type of augmentation technique) and the other 10 are real-values (magnitudes). Since categorical values are involved, I configured the Bayesian optimizer to use a random forest estimator.</p><h4>How DeepAugment finds the best policies</h4><p>The three major components of DeepAugment are the <strong>controller</strong> (Bayesian optimizer), the <strong>augmenter</strong>, and the <strong>child model</strong>, with the overall workflow as follows: the controller samples new augmentation policies, the augmenter transforms images by the new policy, and the child model is trained from scratch by the augmented images.</p><p>A reward is calculated from the child model’s training history. The reward is returned back to the controller, and it updates its surrogate model with this reward and associated augmentation policy (see section “<em>How Bayesian optimization works” </em>below). The controller then samples new policies again and the same steps repeat. This process cycles until the user-determined maximum number of iterations are reached.</p><p>The controller (Bayesian optimizer) is implemented using <a href="https://scikit-optimize.github.io/">scikit-optimize</a> library’s ask-and-tell method. It is configured to use a <strong>random forest estimator</strong> as its base estimator and <strong>expected improvement </strong>as its acquisition function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6PHTImXp32Z83Kw-.png" /><figcaption>Basic workflow of DeepAugment</figcaption></figure><h4>How Bayesian optimization works</h4><p>The aim of Bayesian optimization is to find a set of parameters that maximize the value of the objective function. A working cycle of Bayesian optimization can be summarized as:</p><ol><li>Build a surrogate model of the objective function</li><li>Find parameters that perform best on the surrogate</li><li>Execute the objective function with these parameters</li><li>Update the surrogate model with these parameters and the score of the objective function</li><li>Repeat steps 2–4 until the maximum number of iterations is reached</li></ol><p>For more information about Bayesian optimization, read <a href="https://towardsdatascience.com/a-conceptual-explanation-of-bayesian-model-based-hyperparameter-optimization-for-machine-learning-b8172278050f">this blog</a> explaining it at a high-level, or take a glance at <a href="https://ieeexplore.ieee.org/document/7352306">this review paper</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IElxnjFbp9JEJ6n34L_vPw.png" /><figcaption>A 2D depiction of Bayesian optimization, where x and y axes represent types of augmentation, and the color at point (i,j) represents CNN model accuracy when it is trained with the data augmented by augmentation i and j.</figcaption></figure><h4>Trade-offs of Bayesian optimization</h4><p>Currently, the standard approaches used for hyper-parameter optimization are random search, grid search, Bayesian optimization, evolutionary algorithms, and reinforcement learning, in the order of method complexity. Bayesian optimization is a better choice than grid search and random search in terms of accuracy, cost, and computation time for hyper-parameter tuning (see an empirical comparison <a href="https://mlconf.com/blog/lets-talk-bayesian-optimization/">here</a>). This is due to the fact that Bayesian optimization learns from runs with the previous parameters, contrary to grid search and random search.</p><p>When Bayesian optimization is compared against reinforcement learning and evolutionary algorithms, it provides competitive accuracies while requiring far fewer iterations. Google’s <a href="https://arxiv.org/abs/1805.09501">AutoAugment</a>, for example, iterates 15,000 times in order to learn good policies (which means training the child CNN model 15,000 times). Bayesian optimization, on the other hand, learns good polices in 100–300 iterations. A rule of thumb for Bayesian optimization is making the number of iterations as much as the number of optimized parameters times 10.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ttqWgMJUJzN_WzwOSKecdg.png" /><figcaption>An intuitive comparison of hyper-parameter optimization approaches. Number of plus signs (+) indicates how good the approach is, by comparison category.</figcaption></figure><h3>Challenges and solutions</h3><p><strong>Challenge 1: </strong>Optimizing for augmentation requires a lot of computational resources, since the child model should be trained from scratch over and over. This dramatically slowed down the development process of my tool. Even though usage of Bayesian optimization made it faster, the optimization process was still not fast enough to make development feasible.</p><p><strong>Solutions:</strong> I developed two solutions. First, I optimized the child CNN model (see below), which is the computational bottleneck of the process. Second, I designed augmentation policies in a more deterministic way, making the Bayesian optimizer require fewer iterations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mP6BGXF4DTUaUfmX.png" /><figcaption>Designed child CNN model. It trains in ~30 seconds (120 epochs) with 32x32 images on an AWS p3.2x large instance (Tesla V100 GPU with 112 TensorFLOPS)</figcaption></figure><p><strong>Challenge 2: </strong>I encountered an interesting problem during the development of DeepAugment. During the optimization of augmentations by training the child model over and over, they started to overfit to the validation set. I discovered that my best-found policies perform poorly when I changed the validation set. This is an interesting case because it is different than overfitting, in the general sense, where model weights are overfitting to the noise in the data.</p><p><strong>Solution:</strong> Instead of using the same validation set, I reserved the rest of the data and the training data as the &quot;seed validation set&quot;, and sampled a validation set with 1000 images at each training of the child CNN model (see data pipeline below). This solved the augmentation overfitting problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X7pEUriVjjJ8JwZH_EASQA.png" /></figure><h3>How to integrate into your ML pipeline</h3><p>DeepAugment is published on <a href="https://pypi.org/project/deepaugment/">PyPI</a>. You can install it from your terminal by running:</p><pre>$ pip install deepaugment</pre><p>And usage is easy:</p><pre>from deepaugment.deepaugment import DeepAugment<br><br>deepaug = DeepAugment(my_images, my_labels)<br><br>best_policies = deepaug.optimize()</pre><p>A more advanced usage, by configuring DeepAugment:</p><pre>from keras.datasets import cifar10<br><br># my configuration<br>my_config = {<br>    &quot;model&quot;: &quot;basiccnn&quot;,<br>    &quot;method&quot;: &quot;bayesian_optimization&quot;,<br>    &quot;train_set_size&quot;: 2000,<br>    &quot;opt_samples&quot;: 3,<br>    &quot;opt_last_n_epochs&quot;: 3,<br>    &quot;opt_initial_points&quot;: 10,<br>    &quot;child_epochs&quot;: 50,<br>    &quot;child_first_train_epochs&quot;: 0,<br>    &quot;child_batch_size&quot;: 64<br>}<br><br>(x_train, y_train), (x_test, y_test) = cifar10.load_data()<br># X_train.shape -&gt; (N, M, M, 3)<br># y_train.shape -&gt; (N)<br>deepaug = DeepAugment(x_train, y_train, config=my_config)<br><br>best_policies = deepaug.optimize(300)</pre><p>For more detailed installation/usage information, visit the project&#39;s <a href="https://github.com/barisozmen/deepaugment">README</a> or run the Google Colab notebook <a href="http://bit.ly/deepaugmentusage">tutorial</a>.</p><h3>Conclusion</h3><p>To our knowledge, DeepAugment is the first method utilizing Bayesian optimization to find the best data augmentations. Optimization of data augmentation is a recent research area, and AutoAugment was one of the first methods tackling this problem.</p><p>The main contribution of DeepAugment to the open-source community is that it makes the process scalable, enabling users to optimize augmentation policies without needing huge computational resources*. It is very modular and &gt;50 times faster than the previous solution, AutoAugment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/0*Fr7rY2PI_5iapS18.png" /><figcaption>Comparison of validation accuracies of WideResNet-28-10 CNN model with CIFAR10 images when they are augmented by policies found by DeepAugment, and when they are not augmented. Validation accuracy is increased by 8.5%, equivalent to 60% reduction in error.</figcaption></figure><p>DeepAugment is shown to<strong> reduce error by 60%</strong> for a WideResNet-28-10 model using the CIFAR-10 small image dataset when compared to the same model and dataset without augmentation.</p><p>DeepAugment currently only optimizes augmentations for the image classification task. It could be expanded to optimize for object detection or segmentation tasks, and I welcome your contributions if you would like to do so. However, I would expect that the best augmentation policies are very dependent on the type of dataset, and less so on the task (such as classification or object detection). This means AutoAugment should find similar strategies regardless of the task, but it would be very interesting if these strategies end up being very different!</p><p>While DeepAugment currently works for image datasets, it would be very interesting to extend it for text, audio or video datasets. The same concept is applicable to other types of datasets as well.</p><p>*DeepAugment takes 4.2 hours (500 iterations) on CIFAR-10 dataset which costs around $13 using AWS p3.x2large instance.</p><h4>Acknowledgements</h4><p>I have done this project during my time at the <a href="https://www.insightdata.ai/">Insight Artificial Intelligence Fellows</a> program over a three week period. I give my thanks to program directors <a href="https://twitter.com/mrubash1">Matt Rubashkin</a> and <a href="https://twitter.com/astronomeramber?lang=en">Amber Roberts</a> for their very helpful guidance, and to my technical advisor <a href="https://twitter.com/melissarunfeldt?lang=en">Melissa Runfeldt</a> for helping me problem solve along the way. I thank to <a href="https://twitter.com/astronomeramber?lang=en">Amber Roberts</a>, <a href="https://twitter.com/EmmanuelAmeisen?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor">Emmanuel Ameisen</a>, <a href="https://www.linkedin.com/in/hollyszafarek">Holly Szafarek</a>, and <a href="https://www.linkedin.com/in/andrew-forrester">Andrew Forrester</a> for their suggestions and editing work on this blog post.</p><p><strong><em>Want to advance your career in Data Science and Artificial Intelligence? </em></strong><a href="https://apply.insightdatascience.com/?utm_source=representations&amp;utm_medium=blog&amp;utm_content=bottom"><strong><em>Apply today!</em></strong></a><strong><em> </em></strong><em>Learn more about the </em><a href="http://insightdata.ai/?utm_source=representations&amp;utm_medium=blog&amp;utm_content=top"><em>Artificial Intelligence</em></a><em> program at Insight!</em></p><h3>Resources</h3><p><strong>GitHub:</strong> <a href="https://github.com/barisozmen/deepaugment">github.com/barisozmen/deepaugment</a></p><p><strong>Demo slide deck:</strong> <a href="https://bit.ly/deepaugmentslides">bit.ly/deepaugmentslides</a></p><p><strong>Colab tutorial:</strong> <a href="https://bit.ly/deepaugmentusage">bit.ly/deepaugmentusage</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e87cf692c366" width="1" height="1" alt=""><hr><p><a href="https://medium.com/insight-data/automl-for-data-augmentation-e87cf692c366">AutoML for Data Augmentation</a> was originally published in <a href="https://medium.com/insight-data">Insight</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>