<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Simone Scardapane on Medium]]></title>
        <description><![CDATA[Stories by Simone Scardapane on Medium]]></description>
        <link>https://medium.com/@simonescardapane?source=rss-2e6b05f0ab28------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*a9IzGCN-TIPmJaJL5IhM3w.jpeg</url>
            <title>Stories by Simone Scardapane on Medium</title>
            <link>https://medium.com/@simonescardapane?source=rss-2e6b05f0ab28------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 05:25:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@simonescardapane/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Deep learning from a programmer’s perspective (aka Differentiable Programming)]]></title>
            <link>https://medium.com/data-science/deep-learning-from-a-programmers-perspective-aka-differentiable-programming-ec6e8d1b7c60?source=rss-2e6b05f0ab28------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec6e8d1b7c60</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[Simone Scardapane]]></dc:creator>
            <pubDate>Tue, 08 Jan 2019 16:40:46 GMT</pubDate>
            <atom:updated>2019-01-24T19:59:57.111Z</atom:updated>
            <content:encoded><![CDATA[<h4>Or why neural networks are <strong>not-so-neural</strong> anymore</h4><p>The main lesson from 2018: <strong>deep learning is “cool”</strong>. One of the main reasons is that the basic problems faced by DL are <strong>general enough</strong> to be of interest to an insane amount of disciplines, from computer vision to neural machine translation to voice interfaces. More importantly, DL concepts can also be <strong>derived and studied from a broad range of point-of-views</strong>, which is why you can find machine learning books and articles arising from the fields of <a href="https://web.stanford.edu/~hastie/Papers/ESLII.pdf">statistics</a>, <a href="https://www.amazon.com/Prediction-Learning-Games-Nicolo-Cesa-Bianchi/dp/0521841089">information theory</a>, signal processing, mathematics, and many others.</p><p>Neural networks themselves, the key tool in DL, were born after WWII from a <strong>unique mix</strong> involving neurophysiology, cybernetics, behaviourist psychology, electrical and later computing engineering, and then survived in one form or the other with multiple rebrandings, most notably in the form of “<a href="https://en.wikipedia.org/wiki/Connectionism">connectionism</a>” first and “deep learning” today.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*JZnJedD8SeCyOy2w.jpg" /><figcaption>Source: <a href="https://twitter.com/benblume/status/932226883764342784">https://twitter.com/benblume/status/932226883764342784</a>.</figcaption></figure><p><strong>Practitioners in the last years have witnessed a further rebranding</strong>, that started with the spread of highly-flexible programming libraries such as <a href="http://pytorch.org/">PyTorch</a>, <a href="https://chainer.org/">Chainer</a>, and others. Despite their differences, all these libraries share two fundamental characteristics: (i) back-propagation is fully automatic and transparent with respect to the user (via the use of <a href="http://www.jmlr.org/papers/volume18/17-468/17-468.pdf">automatic differentiation</a>); and (ii) code for the networks and the models’ logic integrate seamlessly with other programming constructs, such as conditional branchings and loops.</p><p>As a result, despite their biological namesake, <strong>neural networks today are more suited to a programmer’s mindset,</strong> and its knowledge of modularity and encapsulation, than to, e.g., biologists and their knowledge of synaptic adaptation and chemical neurotransmitters.</p><p>This shift in perspective goes under many names, most notably “<a href="https://www.nature.com/articles/nphys3272">differentiable programming</a>” (an older idea popularized by <a href="https://www.facebook.com/yann.lecun/posts/10155722686332143">Yann LeCun one year ago</a>) and “<a href="https://medium.com/@karpathy/software-2-0-a64152b37c35">software 2.0</a>”. A differentiable program (DP) is a piece of code implemented using “differentiable” operators (aka neural networks), whose internal logic is adapted by one or more optimization routines in a data-driven fashion. The conceptual difference with “standard” neural networks lies in the diversity of differentiable operators that can be used (e.g, <a href="https://en.wikipedia.org/wiki/Differentiable_neural_computer">differentiable read/write operations</a>, <a href="https://arxiv.org/abs/1808.00508">neural arithmetic units</a>, etc.), and in the fact that these operators are blended with a wide variety of standard constructs and “software 1.0” pieces.</p><blockquote>Differentiable Programming is little more than a rebranding of [modern] Deep Learning techniques […] But the important point is that people are now building a new kind of software by assembling networks of parameterized functional blocks and by training them from examples using some form of gradient-based optimization. (source: <a href="https://www.facebook.com/yann.lecun/posts/10155722686332143">Yann LeCun on Facebook</a>).</blockquote><p>As a corollary, all software design problems, from <a href="https://medium.com/@jonathan_hui/debug-a-deep-learning-network-part-5-1123c20f960d">debugging</a> to <a href="https://en.wikipedia.org/wiki/Adversarial_machine_learning">security</a>, <a href="https://github.com/tensorflow/privacy">privacy</a>, and <a href="https://arxiv.org/abs/1807.10875">fuzzing</a>, can find extensions and their place in the new extended “differentiable software” stack.</p><h4>The purpose of this article</h4><p>More than a rebranding, DP can be seen as a change of mindset: deep networks are not anymore <em>biologically-inspired</em> tools that somehow “learn” as our brain do —<strong> they are programming constructs</strong>. In this line of reasoning, in this article I want to show how many machine learning / deep learning concepts can arise naturally when starting from pure programming, i.e., thinking of unit testing, encapsulation, modularity, and the like. This is not another “<em>deep learning explained to programmers” </em>article — or at least I hope so! This is more a sketch of a “<strong>deep learning explained FROM programmers</strong>” article: the natural step in moving towards differentiable programming.</p><blockquote>Note: this piece is a work-in-progress. Any comment or feedback is highly appreciated.</blockquote><h3>Unit testing and “soft” assertions</h3><p>DPs, just like their standard counterparts, are composed by <em>functions</em>. Functions are (preferably small) pieces of code that takes some input parameters <em>x</em>, and return some output value <em>y</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*yDf2j1XKfMNDj2XZ.png" /><figcaption>Source: <a href="https://tectrick.org/programming-function-definition/function/">https://tectrick.org/programming-function-definition/function/</a>.</figcaption></figure><p>Any program can be expressed as a sequence of elementary operations, <a href="https://en.wikipedia.org/wiki/Structured_program_theorem">interleaved with basic control</a> <a href="https://en.wikipedia.org/wiki/Structured_program_theorem">flow and loops</a>. As any experienced programmer knows, writing a function is much simpler that ensuring that the function is <strong>correct</strong>. This is why in modern programming practice, any function should come with one or more <strong>unit tests</strong>, that evaluate its correct behavior on a series of use cases:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54b06738efbd1ef22f363dbefe63154e/href">https://medium.com/media/54b06738efbd1ef22f363dbefe63154e/href</a></iframe><p>In machine learning terminology, this set of use cases is called the <strong>test set</strong>. An intriguing property of unit tests is that they are generally simpler both to design and to implement than the function they are testing (which is why <a href="https://en.wikipedia.org/wiki/Test-driven_development">an entire programming phylosophy</a> is built on top of them).</p><p>Consider for example a bank’s programmer writing a function to determine whether a given client <em>x</em> should be given a loan (<em>y=1</em>) or not (<em>y=0</em>). The full body of this function will be (probably) a messy thing, full of insider knowledge from his/her part, nested if-else branches, judgmental evaluations, and the like. <strong>Testing the function, however, is trivial</strong>: one can simply select a series of known clients for which the desired output is known, and evaluate the function on them.</p><p>In a standard unit test, scoring badly a single client from our test set will fail the entire routine. However, nothing prevents us from writing <a href="https://automationrhapsody.com/soft-assertions-not-fail-junit-test/"><strong>soft test assertions</strong></a> that evaluate without necessarily failing. For example, we might consider our test successful in this scenario if more than 75% of the clients in the test set are correctly classified from our routine.</p><h3>Soft unit testing allows for optimization?</h3><p>Having soft test measures as described above brings above the idea of <strong>optimizing</strong> our program: a function scoring 80% is probably better than a function scoring 77%, while still leaving open room for improvement. Sadly, we lack a structured way of deciding <strong>how to modify</strong> the function in general.</p><blockquote>Note: we could decide to randomly mutate every single instruction inside our function (e.g., change an “if a &gt; 0” clause to an “if a ≤ 0” clause) and test the new set of functions, a.k.a. <a href="https://en.wikipedia.org/wiki/Genetic_programming">genetic programming</a>. However, we have no guarantee that even one of these possibly <strong><em>thousands</em> </strong>of mutated functions will somehow be better than our original one!</blockquote><p>The insight beyond DPs is that if we suitably restrict the type of elementary operations that we use in our program, then it is possible to design a much better routine for optimizing it from our data. As the name suggests, a DP requires all elementary functions to be <strong>differentiable</strong>, i.e.:</p><ol><li>To change <em>continuously</em> with respect to their input parameters;</li><li>To allow for <a href="https://en.wikipedia.org/wiki/Partial_derivative"><em>derivatives</em></a><em> </em>with respect to the same parameters.</li></ol><p>Let’s see an example of a differentiable function, with a single input parameter, <em>b</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/0*QwSFYw4FzehXoMz1.png" /><figcaption>Source: <a href="https://mathinsight.org/image/derivative_example_function_3_derivative">https://mathinsight.org/image/derivative_example_function_3_derivative</a>.</figcaption></figure><p>The blue line is <strong>continuous</strong>: there are no discontinuities between any value of its input parameters and the ones immediately close. The function also allows for <strong>derivation</strong> (the green line), and we can inspect the derivative to see whether the function will <em>increase</em> or <em>decrease </em>when we increase or decrease the input <em>b</em>. Examples of differentiable functions are the sum, the product, the square root…</p><blockquote>Limiting ourselves to differentiable functions looks like an immense challenge, since any “crisp” decision, resulting in a clear discontinuity in our graph, <strong>will be forever beyond the grasp of our algorithm</strong>. Even taking a simple 0/1 binary decision, or deciding to read a single entry from a database, are instructions that cannot be encoded in such a way. In the AI field, <a href="http://futureai.media.mit.edu/wp-content/uploads/sites/40/2016/02/Symbolic-vs.-Subsymbolic.pptx_.pdf">this is known as the <strong>symbolic vs. subsymbolic</strong> challenge</a>: an approach based on continuous functions cannot manipulate in a straightforward way discrete symbols, such as words or mental categories.</blockquote><p>Apart from the differentiability of f(x), we also need to ensure that the <strong><em>testing </em>procedure is differentiable</strong>. If both constraints are satisfied, we gain an immensely powerful tool in exchange for our lack of expressiveness, in the form of <strong>numerical optimization</strong>. Using information on the derivatives, we can design automatic iterative algorithms that are able to <em>automatically</em> learn the “shape” of our functions by maximizing the test metric. You can think of this as an extension of a compiler, that in addition to producing executable code, “calibrates” the internal parameters of the function that determine how the constituent elementary functions vary.</p><h3>“Information leakage” and overfitting</h3><p>Our discussion up to this point has a big “elephant in the room” that we have not mentioned: numerically optimizing the same performance measure we wish to test is a rather bad idea. In order to see this, consider that by looking at a unit test, we can always implement a simple lookup table mimicking its test cases, that will <strong>always</strong> achieve 100% accuracy in the test. Another way of saying this is that “<em>learning is not memorization</em>”: a bank is only interested in historical information on its clients if it can learn something about <em>future</em> clients, that it has not seen up to now.</p><p>In machine learning, a function that adapts too much to its test data is said to <strong>overfit</strong>. The solution to this problem is straightforward: we simply need to ensure that <strong>the data that we use for optimization is different from the data we use for testing</strong>. Going back to our programming example, any programmer writing both the function and the unit tests is prone to “information leakage” and possibly cheating, but if we use <a href="https://www.testuff.com/why-developers-shouldnt-test-their-own-code/">two different programmers</a> (a.k.a., black-box testing), then we are fully confident our test routines tell us something meaningful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*u1nNQMIzUlad-Z2S.gif" /><figcaption>Black-box unit testing requires the tester to have no knowledge of the underlying program. In DPs, this is achieved trivially by using separate datasets for the optimization phase and the testing phase (source: <a href="http://softwaretestingfundamentals.com/black-box-testing/">http://softwaretestingfundamentals.com/black-box-testing/</a>).</figcaption></figure><p>Training/test separation also has another advantage: while the training criterion has to be differentiable, the test criterion does not have such requirement. Consider again the problem of classifying users: even a very simple objective such as “f(x1) == y1” (something called <strong>the 0/1 loss</strong> in ML terminology) cannot be differentiated, let alone directly optimized. However, we can optimize an appropriate <strong>surrogate</strong> term instead (such as the <a href="https://en.wikipedia.org/wiki/Cross_entropy">cross-entropy</a>), and still be able to test our application in terms of accuracy.</p><h3>The role of the differentiable programmer</h3><p>In “software 1.0”, the programmer chooses a sequence of operations for the function, and then selects a series of use cases for the unit test. In a DP, test cases are more properly <strong>collected </strong>than <strong>chosen</strong>, as they represent historical information from which we want to extract some insights. Because DPs need data also in their optimization phase, and because this data generally needs to be at least one order of magnitude greater than the test cases, differentiable programming is a <em>data-hungry</em> discipline: finding data, labeling it, ensuring that it properly describes our problem domain, eliminating biases are all prerequisites for tackling a software problem in the “2.0” version.</p><p>Next, a programmer needs to choose a correct metric to optimize (the <strong>loss</strong>), and the specific shape for f(x). Just as all the programs currently in execution all around the world are derived from a relatively small number of primitives, <strong>a small number of differentiable operators</strong> allows us to define an exponential number of possible “program structures” for f(x), and the ability to do this intuitively is what distinguishes a deep learning engineer with years of experience from a beginner.</p><p>Many marketing campaigns advocate for deep learning as a tool for <em>automatically </em>solving tasks by throwing more and more data at the problem. Clearly, <strong>this could be not be further from the truth</strong>. The mental role of the programmer is shifting from defining a very precise sequence of symbolic operations, to specifying a template of differentiable operations whose details should be learned from the data. This is not a trivial task, because we are not structured for reasoning in subsymbolic terms, and we find it extremely hard to mentally visualize a sequence of subsymbolic operations. As a result, <strong>building a DP becomes a fascinating mix of intuition, experience, bag-of-tricks, and clever debugging</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/0*17YaK1eqKkU9ujO1" /><figcaption>It looks cool, but only until it doesn’t work and you have no idea what to do with these bunches of numbers (source: <a href="https://bdtechtalks.com/2018/02/27/limits-challenges-deep-learning-gary-marcus/">https://bdtechtalks.com/2018/02/27/limits-challenges-deep-learning-gary-marcus/</a>).</figcaption></figure><p>More in general, it is also difficult to fully isolate the “compiler” (the optimization routine) from the rest of the program, and as a consequence one has to tinker with optimization parameters and flags, or even modify the models themselves to make them easier to optimize.</p><h3>Differentiable routines</h3><p>The prototypical differentiable operation is a <strong>linear combination</strong>. For example, a linear combination of two inputs <em>a</em> and <em>b</em> is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/0*UfBHDWo8pJFI7_PK" /></figure><p>Both <em>a</em> and <em>b</em> concur in deciding the value of c, each with a certain weight that can be found by the algorithm. Linear combinations are the building blocks of all DPs, because they represent the simplest possible ways of inserting parametric relations between objects in our program. <em>c</em> can be an intermediate step in our computation, or even the final output value.</p><p>Mathematically, linear combinations <strong>cannot be stacked</strong>: a linear combination of linear combinations is the same as a single linear combination with different parameters (you probably need to read this twice). If we want to have sequences of linear operations, we need to interleave each of them with one or more <em>nonlinearities</em>. In the terminology of deep networks, these are called <strong>activation functions. </strong>Linear combinations and activation functions are the basics of classical neural networks, loosely representing “synapses” (the connecting lines in the diagram above), and thresholds.</p><p>DPs, however, are not limited to operations that have a biological resemblance. The “DP toolkit” today is very broad, and we can just mention a few representative examples.</p><h4><strong>Differentiable branching and attention</strong></h4><p>Consider a generic conditional instruction of the form “<em>if condition a is satisfied, then do f(x), otherwise do g(x)</em>”. As long as condition <em>a</em> is not a part of the differentiable component of our program, then we have no problem. However, things are harder when we also want to learn (optimize) the clause, because as stated above we cannot differentiate with respect to a binary decision.</p><p>The differentiable version of branching has two components. The first one is to preprocess the condition <em>a </em>with a so-called <strong>sigmoidal function</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/0*IMKVUw_WgpYWOKhy.png" /><figcaption>The sigmoid function (<a href="https://en.wikipedia.org/wiki/Sigmoid_function">taken from Wikipedia</a>).</figcaption></figure><p>As you can see from the graph, irrespective of the value of <em>a</em>, the value of <em>σ(a) </em>(the output of the sigmoid) will be a number between 0 and 1, probably closer to one of the two extremes. Instead of simply selecting one of the two branches, then, a “<em>differentiable if</em>” takes a combination of <em>both branches</em> weighted by the condition itself:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/0*AbHFDBPMA4o11ZDT" /><figcaption>A differentiable equivalent of branching.</figcaption></figure><p>In this way, the condition <em>a</em> can be the output of another DP learned together with the rest. Because of the form of the previous expression, this idea also goes under the name of <strong>gating</strong> in the deep learning literature. A similar mechanism, <strong>attention</strong>, allows to implement differentiable routines for giving <a href="https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html">more or less weight</a> to different components of a program.</p><h4>Differentiable reading and writing</h4><p>Reading and writing from memory is another key component of most programs. Once again, however, if these operations need to be part of a DP, they need to be “smoothed” to make them differentiable (since selecting a <em>single</em> address in memory is a non-differentiable operation). Fascinatingly, one can design differentiable read/write operations using the same logic as before: we let the program read/write <em>all</em> locations, via a suitable linear combination of the elements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*YwlIIJh4wGZ6OFt2kRk22Q.png" /><figcaption>Source: <a href="https://distill.pub/2016/augmented-rnns/">https://distill.pub/2016/augmented-rnns/</a>.</figcaption></figure><h4>Differentiable arithmetic</h4><p>Classical arithmetic is also beyond the scope of DPs, because we have no trivial way of forcing the weights of our program to be exact integers. Nonetheless, we can try to bias our architecture to make <em>arithmetic-like</em> computations. This is the case of the recent <a href="https://arxiv.org/pdf/1808.00508.pdf"><strong>neural arithmetic logic unit</strong></a>, which uses a clever combination of nonlinearities and attention mechanisms to force weights to be close to either -1, 0, or +1.</p><figure><img alt="Image result for neural accumulator" src="https://cdn-images-1.medium.com/proxy/1*vMYerlUvUP5gw4LDZv-aSg.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/1808.00508.pdf">https://arxiv.org/pdf/1808.00508.pdf</a>.</figcaption></figure><h4>Differentiable loops</h4><p>Now consider a generic loop mechanism, where we iterate over the elements of an array <em>a</em> and collect the results through the application of a function <em>f</em>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fa2e34ef40cc51975b91bb48b42db286/href">https://medium.com/media/fa2e34ef40cc51975b91bb48b42db286/href</a></iframe><p>This kind of operation does not require any modification in the differentiable world! In fact, if <em>f</em> is a DP, we obtain a known object called a <strong>recurrent neural network</strong>, a network with a small memory (in the form of the intermediate values <em>z</em>) that can learn to process sequences of elements:</p><figure><img alt="Image result for recurrent neural network" src="https://cdn-images-1.medium.com/proxy/0*WdbXF_e8kZI1R5nQ.png" /><figcaption>Source: <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">http://colah.github.io/posts/2015-08-Understanding-LSTMs/</a>.</figcaption></figure><p>Recurrent networks can also be combined with gates to improve their capabilities, or with small attention mechanisms to automatically learn <a href="https://distill.pub/2016/augmented-rnns/">when to stop iterating</a>.</p><h4>Other differentiable operations</h4><p>Clearly, there are many more differentiable operators! If you are working with special types of data (e.g., images, graphs) you can define specific operations for them, such as convolutional or diffusion operators. Similarly, if you need a more structured output (e.g., sequences), you can combine other operators to bias your DP towards these special cases.</p><h3>Concluding, at last</h3><p>If one really wants to advance deep learning towards differentiable programming, it is not enough to have math tutorials for programmers: <strong>we need to rephrase/rethink the entire field in programming terms</strong>. This post is my “mental sketch” of how such rephrasing could go.</p><p>It is common to describe deep learning in terms of “learning from data”. However, adapting a program to the data at hand is not necessarily a foreign concept for a programmer: a software that fine-tunes its internal logic by reading summaries from a database could be said of performing a very rough version of this. For an experienced programmer, it is much harder conceptually to think of a blend of “crisp” and <em>differentiable</em> programs, and how one can develop an intuition towards <strong>how to solve a problem with a set of primitive differentiable operators</strong>. This one, I believe, will be the main “mental bareer” to the full adoption of a software 2.0 stack.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec6e8d1b7c60" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/deep-learning-from-a-programmers-perspective-aka-differentiable-programming-ec6e8d1b7c60">Deep learning from a programmer’s perspective (aka Differentiable Programming)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multitask learning in TensorFlow with the Head API]]></title>
            <link>https://medium.com/data-science/multitask-learning-in-tensorflow-with-the-head-api-68f2717019df?source=rss-2e6b05f0ab28------2</link>
            <guid isPermaLink="false">https://medium.com/p/68f2717019df</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[convolutional]]></category>
            <dc:creator><![CDATA[Simone Scardapane]]></dc:creator>
            <pubDate>Fri, 04 Jan 2019 08:26:51 GMT</pubDate>
            <atom:updated>2019-01-04T15:09:11.551Z</atom:updated>
            <content:encoded><![CDATA[<h4>An introduction to custom estimators for multitask learning</h4><p>A fundamental characteristic of human learning is that we learn <em>many things</em> simultaneously. The equivalent idea in machine learning is called <strong>multi-task learning (MTL),</strong> and it has become increasingly useful in practice, particularly for reinforcement learning and natural language processing. In fact, even in standard <em>single-task</em> situations, <strong>additional auxiliary<em> </em>tasks</strong> can be devised and included in the optimization process to <a href="https://arxiv.org/abs/1611.05397">help learning</a>.</p><p>This post provides an introduction to the field by showing how to solve a simple multi-task problem in an image classification benchmark. The focus is on an experimental component of TensorFlow, the <a href="https://www.tensorflow.org/api_docs/python/tf/contrib/estimator/regression_head">Head API</a>, that helps in designing custom estimators for MTL, by decoupling the shared component of the neural network from the task-specific ones. Along the route we will also have the opportunity to discuss additional features from the TensorFlow core, including <em>tf.data, tf.image</em>, and the custom estimators.</p><blockquote>The code for the tutorial is available as a <a href="https://colab.research.google.com/drive/1NMB9lpi7P-GkkELkMU0h-yHtUq531D_Z">fully contained Colab notebook</a>, feel free to test and experiment!</blockquote><h4>Content at a glance</h4><p>To make the tutorial more interesting, we consider a realistic use case, by re-implementing (a part of) a 2014 paper: <a href="http://mmlab.ie.cuhk.edu.hk/projects/TCDCN.html">Facial Landmark Detection by Deep Multi-task Learning</a>. The problem is simple: we are given a facial image, and we need to localize a series of <em>landmarks</em>, i.e., points of interest on the image (nose, left eye, mouth, …), and <em>tags</em>, including the age and gender of the person. Each landmark/tag constitutes a separate task on the image, and the tasks are clearly correlated (i.e., think of predicting the position of the left eye knowing first where the right one is).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YFV6paurEwZvnvk4.jpg" /><figcaption>Sample images from the dataset (<a href="http://mmlab.ie.cuhk.edu.hk/projects/TCDCN.html">source</a>). The green dots are the landmarks, and each image is also associated with some additional labels, including age and gender.</figcaption></figure><p>We split our implementation in three parts: (i) loading the images (using <em>tf.data</em> and <em>tf.image</em>); (ii) implementing the convolutional network from the paper (using the <a href="https://www.tensorflow.org/guide/custom_estimators">custom estimators</a> of TF); (iii) adding the MTL logic with the Head API. Lot of things to see, no time to loose!</p><h4>Step 0 — Loading the dataset</h4><p>After downloading the dataset (<a href="http://mmlab.ie.cuhk.edu.hk/projects/TCDCN/data/MTFL.zip">link</a>), a quick inspection reveals that the images are split among three different folders (AFLW, lfw_5590, and net_7876). Train and test splits are provided through different text files, each row of which corresponding to the path of an image and labels:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*ieG6krpHj0rsyRokb2_JDQ.png" /><figcaption>The first image and labels from the training dataset. Blue numbers are image locations (from the top left), red numbers are classes (see below).</figcaption></figure><p>For simplicity we will use Pandas to load the text files and adjust the path URLs to the Unix standard, e.g. for the training part:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f241ad5a4ded81d7d775d130601c616e/href">https://medium.com/media/f241ad5a4ded81d7d775d130601c616e/href</a></iframe><blockquote>Since the text file is not huge, using Pandas in this case is slightly easier and provides a bit of flexibility. For larger files, however, a better choice is to use directly the tf.data object <a href="https://www.tensorflow.org/api_docs/python/tf/data/TextLineDataset">TextLineDataset</a>.</blockquote><h4>Step 1 — Working with tf.data and the Dataset object</h4><p>Now that we have our data, we can load it using <em>tf.data </em>to make it estimator-ready! In the simplest case, we can just slice through the Pandas’ DataFrame to obtain our batches of data:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/80fb01939a3106c06dba779c075d83fc/href">https://medium.com/media/80fb01939a3106c06dba779c075d83fc/href</a></iframe><p>Previously, a major problem of using tf.data with Estimators was that debugging the dataset was rather complex, having to go through <a href="https://www.tensorflow.org/api_docs/python/tf/Session">tf.Session</a> objects. From the latest versions, however, it is possible to debug the datasets with <a href="https://medium.com/tensorflow/building-an-iris-classifier-with-eager-execution-13c00a32adb0">eager execution enabled</a>, even when working with estimators. As an example, we can use the Dataset to build batches of 8 elements, take the first batch, and print everything on screen:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/daf5a14b1b5c6b814233ef9c7b2c572e/href">https://medium.com/media/daf5a14b1b5c6b814233ef9c7b2c572e/href</a></iframe><p>It is now time for loading the images starting from the paths! Note that in general this is not trivial, because images can come in many different extensions, sizes, some in black-and-white, and so on. Luckily for us, we can <a href="https://www.tensorflow.org/guide/datasets#preprocessing_data_with_datasetmap">take inspiration from a TF tutorial</a> to build a simple function to encapsulate all this logic, leveraging the tools in the <a href="https://www.tensorflow.org/api_docs/python/tf/image">tf.image</a> module:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/52d19fc1bb321755d99519bbf64e6bac/href">https://medium.com/media/52d19fc1bb321755d99519bbf64e6bac/href</a></iframe><p>The function takes charge of most parsing problems:</p><ol><li>The ‘channels’ parameter allows to load both colour and b/w images in a single line;</li><li>we resize all images to our desired format (40x40, in accordance with the original paper);</li><li>on line 8, we also normalize our landmark labels to denote a relative location between 0 and 1, instead of an absolute one (since we resized all images, and images might come with different shapes).</li></ol><p>We can apply the parsing function to each element of a Dataset using its internal ‘map’ function: putting this together with some additional logic for training/testing, we obtain our final loading function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f399bd4fe266b6c96758619b3373e04/href">https://medium.com/media/8f399bd4fe266b6c96758619b3373e04/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*7SRuNTfXRXkP2kId4utYpQ.png" /><figcaption>A single image successfully loaded from the dataset.</figcaption></figure><h4>Step 2 — Building a convolutional network with the custom estimators</h4><p>As next step, we want to replicate the convolutional neural network (CNN) taken from the original paper:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*ByvwvM5gWI7T0GR41B2atg.png" /><figcaption>Source: <a href="http://mmlab.ie.cuhk.edu.hk/projects/TCDCN.html">Facial Landmark Detection by Deep Multi-task Learning</a>.</figcaption></figure><p>The logic of the CNN is composed of two parts: the first one is a generic feature extractor for the entire image (which is shared across all tasks), while for each task we have a separate, smaller model acting on the final feature embedding of the image. For reasons apparent below, we will refer to each of these simpler models as an ‘head’. <strong>All heads are trained simultaneously through gradient descent.</strong></p><p>Let us start from the feature extraction part. For this, we leverage the <a href="https://www.tensorflow.org/api_docs/python/tf/layers">tf.layers</a> objects to build our main network:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa2a241a5af644a094c1f5d9a66b9264/href">https://medium.com/media/aa2a241a5af644a094c1f5d9a66b9264/href</a></iframe><p>For the moment, we will focus on a single head/task, i.e., estimation of the nose position in the image. One way to do this is using <a href="https://www.tensorflow.org/guide/custom_estimators">custom estimators</a>, allowing to combine our own model implementation with all the functionalities of a standard Estimator object.</p><p>One drawback of custom estimators is that their code tends to be quite ‘verbose’, because we need to encapsulate the entire logic of the estimator (training, evaluation, and prediction), into a single function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f9319394f0aa7bd6dd1602b5693c389/href">https://medium.com/media/6f9319394f0aa7bd6dd1602b5693c389/href</a></iframe><p>Roughly speaking, the model function receives a <em>mode</em> parameter, that we can use to distinguish what kind of operation (e.g., training), we are expected to do. In turn, the model function exchanges all the information with the main Estimator object via another custom object, an <a href="https://www.tensorflow.org/api_docs/python/tf/estimator/EstimatorSpec">EstimatorSpec</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*YTL-vPk7JH4MZXHve8BNcw.png" /><figcaption>Schematic formulation of custom estimators (<a href="https://school.geekwall.in/p/ByF4_digQ/tensorflow-estimator-dataset-apis">source</a>).</figcaption></figure><p>Not only does this make the code harder to read, but most of the code above tends to be ‘boilerplate’ code, that only depends on the specific task we are facing, e.g., using the mean squared error for a regression problem. The Head API is an experimental feature designed to simplify the writing of the code in this type of situations, and it is our next topic.</p><h4><strong>Step 3a — Rewriting our custom estimator with the Head API</strong></h4><p>The idea of the Head API is that the main prediction component (our model function above) can be generated automatically once a few key items are specified: the feature extraction part, the loss, and our optimization algorithm:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jcoAUGzHQn3_7OYvU58t4Q.png" /><figcaption>Source: <a href="https://www.youtube.com/watch?v=4oNdaQk0Qv4&amp;t=868s">YouTube, The Practitioner’s Guide with TF High Level APIs (TensorFlow Dev Summit 2018)</a>.</figcaption></figure><p>In a sense, this is a similar idea as the high-level interface of <a href="https://keras.io/">Keras</a>, but it still leaves enough flexibility to define a series of more interesting heads, as we will see shortly.</p><p>For the moment, let us rewrite the previous code, this time by using a “regression head”:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5cb1d9037790c4bba5d5390a7dd08a0c/href">https://medium.com/media/5cb1d9037790c4bba5d5390a7dd08a0c/href</a></iframe><p>For all intents and purposes, the two models are equivalent, but the latter is more readable and less prone to errors, as most of the estimator-specific logic is now encapsulated inside the head. We can train either of the two models using the ‘train’ interface of the estimator, and start getting our predictions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*y9L_FSwYp-WQhHbd2cMr8Q.png" /><figcaption>Example of prediction for our single-task model.</figcaption></figure><blockquote>Please do not confuse the Head API (which is in tf.contrib) with <a href="https://www.tensorflow.org/api_docs/python/tf/contrib/learn/Head">tf.contrib.learn.head</a>, which is deprecated.</blockquote><h4>Step 3b — Multi-task learning with the multihead</h4><p>We finally get to the more interesting part of this tutorial: the MTL logic. Remember that, in the simplest case, doing MTL is equivalent to having ‘multiple heads’ on top of the same feature extraction part, as shown schematically here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*MR6HOjYVizjnnsUtmCdshg.png" /><figcaption>Source: <a href="http://ruder.io/multi-task/">An Overview of Multi-Task Learning in Deep Neural Networks</a> (Sebastien Ruder).</figcaption></figure><p>Mathematically, we can optimize all tasks jointly by minimizing <em>the sum</em> of the task-specific losses. For example, assume we have loss L1 for the regression part (mean squared error over each landmark), and L2 for the classification part (different tags), we can minimize L = L1 + L2 through gradient descent.</p><p>After this (quite lengthy) introduction, you might not be surprised that the Head API has a specific head for this situation, called the <em>multi-head</em>. As per our previous description, it allows linearly combining multiple losses deriving from separate heads. At this point, I will let the code speaks for itself:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/26738ab42347dd2c451f28cf56bd183c/href">https://medium.com/media/26738ab42347dd2c451f28cf56bd183c/href</a></iframe><p>For simplicity I am only considering two tasks: the prediction of the nose position, and of the face ‘pose’ (left profile, left, frontal, right, right profile). We just need to define two separate heads (a regression one, and a classification one), and combine them with the <a href="https://www.tensorflow.org/api_docs/python/tf/contrib/estimator/multi_head">multi_head</a> object. Adding more heads now is only a matter of a few lines of code!</p><blockquote>There is a slight modification to the input function that we omitted here for brevity: you can find it on the Colab notebook.</blockquote><p>The estimator at this point can be trained with standard methods, and we can get both predictions simultaneously:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*QebCjcBSH-mMhaVs2fbdOQ.png" /><figcaption>Predictions of our multitask model: node position and pose (frontal in this case).</figcaption></figure><h4>Concluding…</h4><p>I hope you appreciated this small introduction to the MTL problem: if you are interested, I highly suggest <a href="http://ruder.io/multi-task/">this quite informative post</a> by Sebastian Ruder to learn something more about the field. More in general, talking about MTL was the perfect excuse to introduce some interesting concepts from the TF framework, most notably the Head APIs. Don’t forget to play around with the <a href="https://colab.research.google.com/drive/1NMB9lpi7P-GkkELkMU0h-yHtUq531D_Z">full notebook on Google Colab</a>!</p><blockquote>This article appeared originally in Italian on the blog of the <strong>Italian Association for Machine Learning</strong>: <a href="https://iaml.it/blog/multitask-learning-tensorflow">https://iaml.it/blog/multitask-learning-tensorflow</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68f2717019df" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/multitask-learning-in-tensorflow-with-the-head-api-68f2717019df">Multitask learning in TensorFlow with the Head API</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From computer bits to human creativity (and back)]]></title>
            <link>https://medium.com/@simonescardapane/from-computer-bits-to-human-creativity-and-back-39a5e1eb0c7b?source=rss-2e6b05f0ab28------2</link>
            <guid isPermaLink="false">https://medium.com/p/39a5e1eb0c7b</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[art]]></category>
            <category><![CDATA[creativity]]></category>
            <dc:creator><![CDATA[Simone Scardapane]]></dc:creator>
            <pubDate>Wed, 18 Apr 2018 16:03:37 GMT</pubDate>
            <atom:updated>2018-04-18T16:03:37.600Z</atom:updated>
            <content:encoded><![CDATA[<p>On April 5, 2016, at the <a href="https://www.looiersgracht60.org/">Looiersgracht 60 Galerie</a> of Amsterdam, dozens of people and television crews crowded an auditorium to assist at the unveiling of “<a href="https://www.nextrembrandt.com/">The Next Rembrandt</a>”, the culminating event of much anticipation and publicity over the previous weeks. The unveiling immediately became a worldwide trending topic on Twitter, with thousands of users exchanging tweets on the event.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*rT7BYfQBmAQlSSkx." /><figcaption>Visitors watching The Next Rembrandt after its unveiling (credit: <a href="http://www.guneysoykan.com/nextrembrandt/">Guney Soykan</a>).</figcaption></figure><p>On the outside, the unveiled object at the center of all this attention looked like a standard Rembrandt painting. In the precision of its execution, in the way light and color played over its visage, it was a work unmistakably amenable to the Dutch master. Even the most unexperienced person in the audience instinctively recognized that unique painting style.</p><p>Still, something was amiss. In fact, <strong>the painting was not a real Rembrandt at all</strong>, or even a skillful copy. It was a piece of art that was fully computer-generated, the result of an ambitious 18-month project carried by ING Bank, J. Walter Thompson Amsterdam, and Microsoft, with advisors coming from TU Delft, The Mauritshuis and the Museum Het Rembrandthuis. The engineers from this vast team embarked on a colossal project for producing the painting, first analyzing in amazing detail 346 known works from the artist, and then generating a 3D image of uncanny resolution evoking the master in all details, from the choice of the subject to the slightest artistic touch. The result was impressive: after centuries, the spirit of the old genius (or, at the very least, a very significant avatar) lived again in the auditorium.</p><p>With him, a question that has haunted artificial intelligence researchers for decades lingered over the audience: <strong>is there a role for computers in art</strong>? Even more controversial in the era of machine learning, <strong>can we teach a computer how to be an artist</strong>?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cMNY5h_GEgHFE6LP.png" /></figure><blockquote>This feature article is part of a series we are writing to promote the <a href="https://2018.datadriveninnovation.org/">3rd edition of the Data Driven Innovation summit</a>, the Italian conference on data and its applications. If you are interested, come join us for a two-days immersive experience in data-driven technologies — its’s free!</blockquote><h3><strong>Human, machines, and everything in the middle</strong></h3><p>As with all things concerning the boundary between humans and machines, the project gathered an incredible feedback over the following weeks, sparking thousands of articles and videos worldwide and earning the companies <a href="http://www.guneysoykan.com/nextrembrandt/">millions in media revenue</a>. In the meantime, it reignited public interest on computer-generated art and its artistic value.</p><p>For many people, machines will never exhibit creativity, by definition. “<em>A computer can only do what we program it to do</em>” is in fact a very old maxim, something that can be traced back to the XIX century mathematician <a href="https://en.wikipedia.org/wiki/Ada_Lovelace">Ada Lovelace</a>, daughter of Lord Byron. For others, like the British-born computer scientist <a href="https://en.wikipedia.org/wiki/Simon_Colton">Simon Colton</a>, it is instead only a <strong>matter of complexity</strong>. According to him, machines’ output could, in principle, become so complex and impossible to fathom for a single person, to be worthy of the adjective “creative”. Colton, who is a full professor at the University of London, is considered one of the pioneers of the field of <a href="https://en.wikipedia.org/wiki/Computational_creativity">computational creativity</a>, a broad scientific endeavor with the aims of pushing the boundaries of artificial imagination as far as possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*Zsc__s6imP56oEnO.png" /><figcaption>When translating an article from the Italian mathematician Luigi Menabrea, Ada Lovelace noted that “[the machine] has no pretensions whatever to <em>originate</em> anything. It can do <em>whatever we know how to order it</em> to perform.” This will go down in history as “Ada’s objection” to AI (credit: <a href="http://blog.stephenwolfram.com/2015/12/untangling-the-tale-of-ada-lovelace/">blog.stephenwolphram.com</a>).</figcaption></figure><p>While this is a debate worthy of discussion, it is not what I want to focus on here. This article is instead about something slightly different. In fact, I do not even want to argue whether a machine can be considered an artist, or whether this is a privilege that will be forever a human prerogative. I want us to consider a trickier question: can <strong>machines and humans together</strong> become <em>better</em> artists?</p><p>For me, what the audience present at Amsterdam that day witnessed was not about artificial intelligence at all. It was about the incredible <strong>sleeping potential that data and computing power have to ignite and revolutionize the art world</strong>, with teams of creatives and engineers working together to unlock imagination in ways that were undreamt of before. It was, as we will argue in this article, machine learning finally transcending its computer science bounds and becoming a workable tool in the hands of makers. What we can rightfully call “data-driven creativity”.</p><h3><strong>Endless variations in a world of algorithms</strong></h3><p>Of course, computer-generated art is not a new concept. Machines have helped us create things since they came into existence (so much so that the <a href="http://www.vam.ac.uk/content/articles/a/computer-art-history/">Victoria &amp; Albert Museum</a> has an entire collection devoted to the topic). As a famous example, Harold Cohen, a British artist who passed away in the same year the “new” Rembrandt was shown to the public, spent decades of his life building AARON, a powerful software-programmed machine able to generate realistic paintings.</p><p>While the Rembrandt was 3D-painted from a digital design, most of AARON’s output was obtained by executing real brushes on a canvas <a href="http://www.computerhistory.org/atchm/harold-cohen-and-aaron-a-40-year-collaboration/">through sophisticated (mechanical) mechanisms</a>. Started as an experiment in what constituted a drawing, the abilities of AARON increased over the years with the increase in ability of its creator to program it better and better. Many of its creations have been exposed in galleries and exhibitions around the world, most notably at the <a href="http://www.tate.org.uk/art/artists/harold-cohen-925">TATE museum</a> and in San Francisco.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/0*qx5L5apgeMuxOxBa.jpg" /><figcaption>A painting from AARON during one of its “naturalistic periods” (credit: the <a href="http://www.aaronshome.com/aaron/aaron/gallery/index.html">official AARON website</a>).</figcaption></figure><p>Much has been written on whether AARON can, and to what extent, be considered an artist. Again, this is not the point we want to make here: Harold Cohen itself has always been careful in being non-committal to this notion. At the same time, he liked to remark that <strong>AARON was (and always remained) a much better painter that he himself could have been</strong>. Neither of them were very good on their own, but together, the human and the machine created something remarkable. Machines can only do what we program them to do, sure, but what we can program them to do is not as simple as we like to think.</p><p>To stress this point, consider another famous creation in the world of computer-generated art, <a href="http://www.thepaintingfool.com/index.html">The Painting Fool</a>. This computer software is the brainchild of another decade of work from <a href="https://en.wikipedia.org/wiki/Simon_Colton">Simon Colton</a>, the computer scientist whom we already mentioned before. While being similar to AARON, The Painting Fool is also able to extract clues from its environment, most notably news articles, to generate its works, such as creating impressive war collages based on the analysis of <a href="https://newatlas.com/creative-ai-algorithmic-art-painting-fool-aaron/36106/">news from the war in Afghanistan</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*_G19PsWS_W-rAlw_.png" /><figcaption>An example of output from The Painting Fool (credit: <a href="http://www.thepaintingfool.com/about/index.html">thepaintingfool.com</a>).</figcaption></figure><p>Both AARON and The Painting Fool show the amazing capabilities of computers to generate endless variations while playing around from a template. Nevertheless, the latter also shows something more, the main idea we want to discuss here: the ability of some computerized art to <em>learn</em> from data, and not just to follow a preprogrammed course of action. In a world where data literally floods us, <strong>adapting from it becomes the key to unlocking the potential of artificial creativity</strong>.</p><h3><strong>Interlude: machine learning and deep neural networks</strong></h3><p>Powering any data-driven technology, of course, are machine learning algorithms, which, as most of the things we discuss here, have a long and fascinating history. Computer scientists have dreamed of computers having the power to automatically extract knowledge from data since the turn of the century, dating back to cybernetics and the beginning of artificial intelligence.</p><p>I already discussed the rise in machine learning interest over the last few years in an <a href="http://2017.datadriveninnovation.org/scardapane-what-is-machine-learning-and-why-you-should-care/">article for the previous edition of the conference</a>. Most of this interest has been driven by the arrival of “deep learning”, a broad term describing a set of algorithms (most commonly in the form of so-called “artificial neural networks”), that have allowed to tackle machine learning problems considered impossible up to that point. <strong>Deep learning has revolutionized many fields</strong>, attracting billions of dollars of investments from companies around the world: whenever you hear “artificial intelligence” in the news, chances are you are just watching a different application of deep learning techniques. Results range from near-human image recognition capabilities (that first showed the power of these techniques) to defeating world champions at Go.</p><p>Deep learning can process complex inputs that, up to that point, were considered the domain of human sensory processing and required complicated engineering algorithms to make them accessible to computers: most notably images, videos, unstructured text, and audio. These are all domains where the information is encoded in <strong>millions of tiny pieces of content</strong> (think of the pixels in an image), and their understanding requires the <strong>simultaneous processing of all of them</strong>. Deep neural networks achieve this aim by learning <em>multi-layered</em> representations that extract the relevant information in multiple processing steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/0*BuoBr6JmZTTBp_Zk.jpg" /><figcaption>A neural network learning to play Atari games (from <a href="https://www.nature.com/articles/nature14236">Human-level control through deep reinforcement learning</a>). This was one of the first works showing that algorithms can learn to play games directly from the raw video input, a task that was considered nigh impossible up to that point.</figcaption></figure><p>What we see with projects like The Next Rembrandt is that <strong>deep learning technologies are quickly moving beyond their engineering bounds</strong>. The combination of deep neural networks and computational creativity is explosive: the possibility for creatives of using algorithms that can automatically process audio, videos, and texts opens up a combinatorial growth and sophistication in what is achievable by computational means. Furthermore, much of this benefits from the fact that most tech companies are now pushing for <strong>open sourcing their codes and their implementations</strong>, making them accessible to a much vaster audience.</p><h3><strong>Creativity in the era of deep learning</strong></h3><p>Let me describe two examples of the power of this combination (algorithms, openness, and creativity), both originating in Google research projects.</p><p>The first one is <a href="https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html">Inceptionism</a>, also known as DeepDream. DeepDream was a technique that a team of researchers from Google developed with the aim of extracting useful knowledge about the mechanisms of deep neural networks, to understand better what it is that these networks learn from data. Since different parts of the network can in principle specialize in recognizing different concepts, the idea was to create artificial images that maximally “activate” one or more of these parts, to visualize what the networks had learnt.</p><p>To the surprise of the researchers, the result were sometimes mesmerizing images, with absurdly vivid colors mixed to impossible superposition of objects, that were described as “the dreams of neural networks” or (for the more comically inclined) “neural networks under LSD”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f7v4CicVI8BIc8_4.png" /><figcaption>Example of images resulting from the DeepDream technique (taken from the <a href="https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html">official Google Research blog post</a> describing the algorithm).</figcaption></figure><p>The idea could have been a simple research curiosity, if not for the fact that the dreamy quality of the generated images was so curious that <strong>it attracted the attention of thousands of people around the world</strong>, who wanted to exploit this effect for much more than just understanding the neural networks. Thanks to the open source code, DeepDream was soon incorporated in hundreds of demos, applications, and projects, from <a href="http://psychic-vr-lab.com/deepdream/">VR interfaces</a> to <a href="https://www.youtube.com/watch?v=dJ1VorN9Cl0">song music videos</a>, making it one of the most popular manias of 2015.</p><p>While the temptation is to discard DeepDream as one of the many quirks of the Internet, you should not be so quick in dismissing it as a joke. It is a perfect example of what happens when you mix a state-of-the-art data-driven technique, make it accessible to the public, and stir it up with human imagination: the results can go much beyond what its creators could have believed possible.</p><p>If you want a more serious example of deep learning applied to art, take a look at <a href="https://magenta.tensorflow.org/">Google Magenta</a>, another research project from the US tech giant. The aim of Magenta is to combine the latest innovations in deep learning with the insights of musicians and the computing power of Google’s infrastructure to generate music of incredible sophistication. Just like DeepDream, all of the Magenta’s results are <a href="https://magenta.tensorflow.org/demos">available to play around with</a>, from demos to code.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FAq3370Prbi4%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DAq3370Prbi4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FAq3370Prbi4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/599d0cb326b2e8427dfcf2e3ec2e47d2/href">https://medium.com/media/599d0cb326b2e8427dfcf2e3ec2e47d2/href</a></iframe><h3><strong>Computer-powered creativity: the next frontiers</strong></h3><p>Just as deep learning gave new life to machine learning, it is revitalizing the idea of using computers and data to help us create things. Today, <strong>the idea that computer art is a niche for a selected group of people proficient in art, computers and math is not true anymore</strong>. Machine learning is seen as such a transformative technology that companies like Microsoft and Google are giving their best to make it “democratic” and accessible to everyone. Similarly, projects like <a href="https://ml4a.github.io/">machine learning for artists</a> or the <a href="https://ami.withgoogle.com/">artists and machine intelligence residency program</a> at Google were created precisely for making data-driven technologies more accessible to an artistic audience.</p><p>Are we just scratching the surface of what is feasible with such technologies? This is the last question I would like to explore in this brief overview. To this end, allow me to make two final remarks on ways that could vastly expand the power and reach of data-driven creativity in the future.</p><p>First, up to now we discussed algorithms that can learn from data stored on a disk, like images or audio. However, we humans are able to adapt to a much wider range of information, including <em>human</em> feedback and the actions of other people. Researchers have tried for a long time to include this kind of feedback inside data-driven algorithms, with techniques having names like multi-armed bandits, reinforcement learning, and interactive evolutionary computation.</p><p>Today, we are starting to see a fascinating mix of these methods with deep learning at work to slowly revolutionize entire fields such as advertising. Companies such as Netflix can now offer <strong>personalized advertising</strong> (e.g., <a href="https://medium.com/netflix-techblog/artwork-personalization-c589f074ad76">in the form of artworks and covers</a>), whose visuals are adapted<em> </em>for each subscriber based on their feedback: a fascinating form of data-driven “micro-art” that has wide implications going far beyond advertising.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*Ie2TecVCUFlyIQPb." /><figcaption>An example of personalized advertising taken from <a href="https://medium.com/netflix-techblog/artwork-personalization-c589f074ad76">the Netflix’s blog</a>. Data-driven algorithms with a human feedback can select different covers for the same movie depending on the preferences of each user (in this example, comedy or romance).</figcaption></figure><p>Painters, makers, advertisers: can data-driven creativity also change other artistic fields, like design and architecture? In order to answer this question, let me go back to our description of deep learning: a technology for automatically generating hierarchies of representations from raw data. A fascinating insight of the last years has been that these representations are not only useful for the task at hand (e.g. recognizing objects in an image), but they can be exploited to <strong>manipulate concepts at higher levels of abstraction</strong>. One of the most famous examples of this is the <a href="https://arxiv.org/abs/1508.06576">neural artistic style transfer</a>, a technique that allows mixing the “style” of a picture with the content of another one. This is obtained by combining the internal “style” and “content” representations learnt by a neural network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m9wAYpnEA2S4553M.png" /><figcaption>Example of neural style transfer (taken from <a href="https://codelabs.developers.google.com/codelabs/tensorflow-style-transfer-android/index.html?index=..%2F..%2Fio2017#0">an official Google Codelab</a> on the topic).</figcaption></figure><p>Can we apply similar ideas to computational art and design? Imagine a sophisticated photo editor that, instead of allowing you to play around with pixels and patches in an image, has buttons to change and manipulate some abstract qualities of your photo, like its style, its “<em>melancholiness</em>”, and so on. This is far from a future technology: similar ideas are being actively developed under the proposed name of <a href="https://distill.pub/2017/aia/">artificial intelligence augmentation</a>. Once again, the problem is not in the technology itself, but in designing user interfaces and high-level abstractions allowing non-technical people to <a href="https://towardsdatascience.com/des-ai-gn-augmenting-human-creativity-with-artificial-intelligence-bb6ff611fa2c">leverage their power</a>.</p><h3><strong>The sleeping power of data-driven imagination</strong></h3><p>Not much of what we discussed here is new. Machine learning, computer art, and user interfaces have existed, in one form or the other, since we invented the first computers. However, the potential of mixing them together has always been relegated to a very specialized audience that wanted to explore the philosophical limits of computer creativity, or who had the technical capacity of developing highly sophisticated machines like AARON.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*Mic0QeSW1LOtc0qPTkhUaA.gif" /><figcaption>A neural network drawing an abstract representation of an electric fan (taken from <a href="https://medium.com/artists-and-machine-intelligence/perception-engines-8a46bc598d57">Perception Engines</a>, one of the projects at the <a href="https://ami.withgoogle.com/">Google’s AMI program</a>).</figcaption></figure><p>What we are witnessing today is a complete change, both in the scale of what is feasible, and in the number of people that have access to these technologies. Pushing for openness and democratization is allowing thousands of developers and artists to have access to the latest deep learning tools for their projects, which are quickly moving from including object recognition in Android applications to having generative data-driven technologies in art installations around the world, manipulating abstract concepts, and including personalized feedback from the viewers. <strong>This is affecting all fields were creativity is valued</strong>, from advertising to design and art. How much further computers and deep learning can change these fields will depend on how good we will be at making them accessible to everyone.</p><blockquote>If you want to learn more about data-driven creativity and its far-reaching implications, come join us and discuss them at the <a href="https://2018.datadriveninnovation.org/">third edition of the data driven innovation summit in Rome</a>!</blockquote><h3>About the author</h3><p><a href="http://ispac.diet.uniroma1.it/scardapane/">Simone Scardapane</a> is a post-doc researcher in Rome, specializing in machine learning. He is also co-founder of the <a href="https://iaml.it/">Italian Association for Machine Learning</a>, a no-profit organization devoted to promoting machine learning in Italy.</p><p><em>Thanks to </em><a href="https://github.com/picorana"><em>Sara Di Bartolomeo</em></a><em> for feedback on a previous draft of this article.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39a5e1eb0c7b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an Iris classifier with eager execution]]></title>
            <link>https://medium.com/google-developer-experts/building-an-iris-classifier-with-eager-execution-3d77ef2c024?source=rss-2e6b05f0ab28------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d77ef2c024</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <dc:creator><![CDATA[Simone Scardapane]]></dc:creator>
            <pubDate>Fri, 13 Apr 2018 18:52:03 GMT</pubDate>
            <atom:updated>2018-04-26T11:39:41.153Z</atom:updated>
            <content:encoded><![CDATA[<h4>(Reposted from the <a href="https://medium.com/tensorflow/building-an-iris-classifier-with-eager-execution-13c00a32adb0">TensorFlow blog</a>)</h4><p>One of the new additions to TensorFlow in the last months has been the <a href="https://research.googleblog.com/2017/10/eager-execution-imperative-define-by.html">eager execution</a>, an additional low-level interface promising to make development a lot simpler and easier to debug. Whenever eager execution is enabled, operations are executed immediately, instead of having to go through a separate execution step. In a nutshell, this means that writing TF code can be (potentially) <strong>as simple as writing pure NumPy code</strong>!</p><p>Being a long-requested feature, I was looking forward to test it. After playing around for a while with the new interface, I can say the new workflow feels much simpler and intuitive, all the methods are integrated almost perfectly with the rest of TF and, despite being in an experimental stage, most of the functionalities are already stably implemented.</p><p>At the same time, eager execution is a further addition in a framework that is already complex for many newcomers. Which is the reason for this post: a guided tutorial for navigating into this new interface by showing all the steps needed to <strong>build a simple classification model</strong> from scratch on the <a href="https://archive.ics.uci.edu/ml/datasets/iris">Iris dataset</a>. At the end of this article, you will have implemented a fully working model in eager execution from which you can continue experimenting.</p><p>This post is also intended as a <strong>self-contained introduction to the eager execution</strong>: if you are new to low-level programming in TensorFlow, you will learn how you can implement your own algorithms and run them using this new interface. But if you already consider yourself an expert in TF, this post will also highlight the differences when enabling eager execution<em> </em>as opposed to the standard graph construction.</p><p>Everybody is ready? Let’s go!</p><p><em>Note: if you want to experiment with eager execution, I also wrote a </em><a href="https://github.com/tensorflow/workshops/blob/master/extras/eager/eager-tutorial-simone.ipynb"><em>Jupyter notebook</em></a><em> to serve as an extended companion to this post. And if you don’t have a working TensorFlow installation? Remember you can run the notebook using the (completely free) </em><a href="https://colab.research.google.com/"><em>Google Colaboratory service</em></a><em> in the cloud!</em></p><h3>Why do we need eager execution?</h3><p>Nowadays TF includes several high-level APIs, such as <a href="https://www.tensorflow.org/programmers_guide/estimators">Estimators</a> and <a href="https://keras.io/">Keras</a>. But many times, you’d also like to explore and run some low-level code, or dig deeper in the framework. At that point, you may like to write or run something simple to begin with.</p><p>To see how the eager execution works, let’s start by adding and multiplying some numbers, and see how the code would look like <strong>without eager execution</strong>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dda8889daf837256b4395bb10ed7209e/href">https://medium.com/media/dda8889daf837256b4395bb10ed7209e/href</a></iframe><p>If you (like me) have been here since the <a href="http://deeplearning.net/software/theano/">early days of Theano</a>, it is hard to remember just <strong>how strange</strong> this code feels if you are not used to it: why do you even need an external object just to print a number? And why can’t you find the value of b <em>anywhere</em>? The reason is that, internally, TensorFlow is building a computational graph describing all the operations, which is then compiled and optimized behind the scenes — optimization that, in the initial development of TF, was supposed to happen only after the full specification of the model.</p><p>Now compare the same example, but after enabling the eager execution:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ac1d797eef966f0a603e59f00a480b7/href">https://medium.com/media/0ac1d797eef966f0a603e59f00a480b7/href</a></iframe><p>This is pretty much NumPy code! In programming terms, the default API for TF is <strong>declarative</strong>: execution only happens when we request the output of a specific node, and it only returns that particular result. The new eager execution, instead, is <strong>imperative</strong>: execution follows definition immediately, with all of its benefits, immediate visual debugging for one. Let’s see what this means for the optimization of our models — but first, let’s take a small detour to see how you can install and enable eager execution.</p><h3>Enabling eager execution in your code</h3><p>Eager execution has been included as an experimental feature since TF v1.5, and for the purposes of this post I will use the latest v1.7rc0 release (mentioning where you might need to change the code for backward compatibility).</p><p>If you want to play with the latest version, the recommended way is to set up a separate virtual environment (you can find detailed instructions on the notebook and on the <a href="https://github.com/tensorflow/tensorflow">official guide</a>) and install the nightly build of TF from the console:</p><blockquote>pip install tf-nightly[-gpu]</blockquote><p>Eager execution can be enabled with a single line of code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ca97a11d26822fbb84f380d17a79c86/href">https://medium.com/media/0ca97a11d26822fbb84f380d17a79c86/href</a></iframe><blockquote>If you are working with v1.5 or v1.6, change <em>tf.enable_eager_execution()</em> with <em>tfe.enable_eager_execution()</em>.</blockquote><p>Importantly, you need to run the command <strong>at the beginning of the program</strong>, otherwise the command will throw an error and refuse to execute. The reason is that, by enabling eager execution, you are changing the default inner mechanisms for TF described before, meaning that several low-level commands will be available only in one of the two modalities (graph construction or eager execution) as described in the documentation.</p><h3>What we will do in this tutorial</h3><p>In the rest of this tutorial, we are going to build a classification model (with eager execution enabled) for the Iris dataset, a simple classification dataset which you should already have encountered when following the <a href="https://www.tensorflow.org/get_started/premade_estimators">starting guide for TF</a>. The task is to classify a set of Irises in three different classes, depending on four numerical features describing their geometrical shape:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*xOxhDM-2Cz8RL16rkRBPKQ.jpeg" /><figcaption>The three classes of Irises we want to classify (<a href="https://www.tensorflow.org/get_started/premade_estimators">taken from Getting Started with TensorFlow</a>). From left to right, original copyright by by <a href="https://commons.wikimedia.org/wiki/User:Radomil">Radomil</a>, CC BY-SA 3.0, <a href="https://commons.wikimedia.org/wiki/User:Dlanglois">Dlanglois</a>, CC BY-SA 3.0, and <a href="https://www.flickr.com/photos/33397993@N05">Frank Mayfield</a>, CC BY-SA 2.0.</figcaption></figure><p>For the purpose of this article I am simply loading the dataset’s version from the <a href="http://scikit-learn.org/">scikit-learn library</a>, normalizing its inputs, and splitting it in two for training and testing. If you are curious, here is the full code for loading the dataset:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/66f3c23362e16e555547b4261465e9cc/href">https://medium.com/media/66f3c23362e16e555547b4261465e9cc/href</a></iframe><p>We will proceed in five steps. The first part is just to get some familiarity with the basic eager objects such as variables and functions, and how they interoperate with NumPy arrays. Then, we will learn how to build the classification model (part 2), optimize it (part 3), load the data in TF (part 4), and debug the overall process (part 5).</p><h3>Building the classifier (1/5): functions, gradients, and variables</h3><p>The basic operation provided by eager execution is <strong>automatic differentiation</strong>: given some function operating on tensors, we can compute the gradient with respect to one or more of these tensors automatically (and, it goes without saying, efficiently).</p><p>If you are costructing a TF graph, functions and gradients are defined as nodes inside the computational graph. With the eager execution enabled, instead, <strong>both of them are standard Python functions</strong>. For example, here is some code defining a simple squaring function and computing its gradients:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1dc7b03fa48538213ee6a308b89f689/href">https://medium.com/media/c1dc7b03fa48538213ee6a308b89f689/href</a></iframe><p>Instead of having to build the graph, all operations are internally saved inside a “tape”, transparently to the user. The tfe.gradients_function is where the magic happens: you can pass any Python function (having as input at least one tensor) as argument, and it will return <strong>another Python function</strong> for computing the gradient with respect to any of its arguments.</p><blockquote>You might have noticed that in line 6 above, we use <em>tf.constant</em> to build a constant tensor: this step is optional, since passing a NumPy array or a float value would trigger an automatic casting operation.</blockquote><p>You can also chain the calls to get higher-order derivatives:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ee874e464340d8324881092e4225bd7/href">https://medium.com/media/1ee874e464340d8324881092e4225bd7/href</a></iframe><p>A fundamental building block of TF are <a href="https://www.tensorflow.org/programmers_guide/variables">variables</a>: stateful arrays of numbers that compose our models and that we need to optimize in the training phase. Eager has a custom implementation of variables:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7c6a0ff055341b0550b40b8b1490dac/href">https://medium.com/media/e7c6a0ff055341b0550b40b8b1490dac/href</a></iframe><p>As you can see, variables in eager are <strong>fully interoperable with NumPy arrays</strong>, and their value is automatically initialized as soon as they are requested (while with graph construction you would need to explicitly call an initialization routine). Using tfe.implicit_gradients , you can also automatically compute the gradients of a function with respect to <em>all</em> variables it uses. For example, this is equivalent to the gradient computation above, but using variables and implicit gradients:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2bad62fc732b83ad9978bcb838c16b3c/href">https://medium.com/media/2bad62fc732b83ad9978bcb838c16b3c/href</a></iframe><p>Now that we have all the building blocks in hand, we can move to a higher level of abstraction, starting with the definition of our neural network.</p><h3>Building the classifier (2/5): layers and networks</h3><p>In theory, we already have everything we need for optimizing a model: we can instantiate a bunch of variables (the parameters of our models), and encapsulate the logic of the model itself inside a function. For example, this is a one-dimensional logistic regression model:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e221076be0b0a4dda7399c271e1dc0cc/href">https://medium.com/media/e221076be0b0a4dda7399c271e1dc0cc/href</a></iframe><p>However, as soon as we start adding more operations (e.g., new hidden layers, dropout, …) this approach does not scale. A simple way to build more complicated models is to use <a href="https://www.tensorflow.org/tutorials/layers">layers</a>, which should be quite familiar to anyone using the high-level interface of TF:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3db2a636548ca145c291279b241e57b8/href">https://medium.com/media/3db2a636548ca145c291279b241e57b8/href</a></iframe><p>In the above example, we are building a model with a single hidden layer and a dropout operation in the middle: the <strong>definition of variables is taken care of inside each layer</strong>, and we can define our model by chaining together the layers inside a simple (once more, Python) function. The model is quite simple to interpret, even if you are not used to TF, and we can implement the dropout logic by just passing another Boolean flag to the function, adding to the readability.</p><blockquote>The <em>tf.layers</em> module is an implementation of <em>tf.keras.layers</em> , which contains a set of high-level layers defined inside the <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers">Keras library</a>. The implementation at this moment is only partial, with new layers being added continuously: in the meantime, you can also use directly the layers defined in <em>tf.keras.layers</em> .</blockquote><p>Since building networks in this fashion is a common workflow, TF also provides a further level of abstraction with the Model object, a custom class that encapsulates the model’s logic, providing a few advanced tools for debugging and checkpointing the training:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98e4757f0e54d53fda1538558b13eca2/href">https://medium.com/media/98e4757f0e54d53fda1538558b13eca2/href</a></iframe><blockquote>If you are working with v1.5 or v1.6, you need to use <em>tfe.Network</em> instead of <em>tf.keras.Model</em> , and you need to explicitly call <em>self.track_layer()</em> for each layer added in the initialization.</blockquote><p>A major benefit of networks is that they encapsulate all the logic concerning the variables of the model. To understand it, let’s play around with our model a little bit:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b5298c8c6fe542c78e6f002535f2329c/href">https://medium.com/media/b5298c8c6fe542c78e6f002535f2329c/href</a></iframe><p>Line 2 is just the initialization of the model. The fifth line shows the <strong>lazy nature </strong>of variables inside a model: even though eager execution has a dynamic workflow, variables are only initialized when a model is run for the first time, because prior to that eager has no indication on the shape of its input tensors. Predictions can be obtained by simply calling the object with some data (line 8): after this, variables are correctly initialized, as shown in line 11.</p><h3>Building the classifier (3/5): losses and optimizers</h3><p>Since we are going to use our model to do some classification, we need to define a proper cost function to train it. Similarly to before, we can use the standard components of TF, by <strong>encapsulating them in a Python function</strong>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d63945fea97386be94f4cb653989a847/href">https://medium.com/media/d63945fea97386be94f4cb653989a847/href</a></iframe><blockquote>As a quick sidenote, we are using <em>softmax_cross_entropy_with_logits_v2</em> , as the old <em>softmax_cross_entropy_with_logits</em> will get deprecated soon! The two are entirely identical here, the difference being that the new version allows for gradient computation with respect to the labels (not needed for this simple example). By removing the one-hot encoding, we can also use <em>sparse_softmax_cross_entropy</em> equivalently.</blockquote><p>For the optimization, we can define our custom logic or (more probably) use one of the <a href="https://www.tensorflow.org/api_guides/python/train">many optimizers already defined inside TF</a>. At this point, you will probably guess what our code will look like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e81e13096d7e2c95301cf58aaddbbd93/href">https://medium.com/media/e81e13096d7e2c95301cf58aaddbbd93/href</a></iframe><p>The only syntactic sugar here is using an anonymous function with no arguments to call the minimization routine: this is needed because the optimizers were designed to work for the graph construction, and have been customized to also support the new eager interface. Another possibility is to explicitly compute the gradients of our model, and use the apply_gradients function of the optimizer: if you are interested I describe this alternative way in the notebook.</p><h3>Building the classifier (4/5): loading data and iterators</h3><p>If you have kept up-to-date with TF, you will know that the preferred way to feed data to our models is though the <a href="https://www.tensorflow.org/programmers_guide/datasets">high-level Dataset API</a>, which can be used to load data from multiple sources and connects with all the high-level APIs. A nice feature of eager execution is that you can encapsulate datasets inside a Pythonesque iterator, and use the iterator to <strong>cycle over it as you would any other iterator</strong>.</p><p>Let’s loop over the training portion of the Iris dataset we loaded before using an iterator:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c1afd1ef27ff16f37b57261c4e4d1e7/href">https://medium.com/media/6c1afd1ef27ff16f37b57261c4e4d1e7/href</a></iframe><p>In the above code we are loading the data inside the Dataset object (line 2), and then using the iterator to loop over mini-batches of 32 elements, computing the average number of positive labels for each batch, e.g.:</p><blockquote>Percentage of class [0]: 31.25 % <br>Percentage of class [0]: 34.375 % <br>Percentage of class [0]: 37.5 % <br>Percentage of class [0]: 25.0 %</blockquote><h3>Building the classifier (5/5): Debugging tools</h3><p>The last thing we need for our task are some tools for debugging and visualizing the results. Because in the standard graph construction the execution is masked underneath the session’s dynamics, debugging has always been a difficult point for newcomers to TF, and the main reason why the <a href="https://www.tensorflow.org/programmers_guide/summaries_and_tensorboard">TensorBoard</a> was developed. In eager execution, however, everything runs dynamically and you can use <strong>any tool in your Python’s toolbox</strong> for it: from Matplotlib’s plots to the console’s outputs, you name it.</p><p>Eager has also a few utilities to simplify the debugging. For example, <a href="https://www.tensorflow.org/versions/master/api_docs/python/tf/contrib/eager/metrics">metrics</a> can be used to compute and accumulate values across each epoch:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef3f0403891bea44f5db1e518630f72f/href">https://medium.com/media/ef3f0403891bea44f5db1e518630f72f/href</a></iframe><p>The code above loops through a single epoch, and for each batch it computes the accuracy of the model. As we have not trained it yet, the accuracy will probably hoover around 33%, or random chance.</p><p>And if you are fond of the good ol’ TensorBoard, you can save values for visualization on the dashboard using the <a href="https://www.tensorflow.org/versions/master/api_docs/python/tf/contrib/summary">experimental implementation of the summaries</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/58af014c22dd1a8202f311589da92417/href">https://medium.com/media/58af014c22dd1a8202f311589da92417/href</a></iframe><p>The syntax here is slightly more involved: after creating a writer, we need to set it as the default one <em>and</em> choose an interval for saving the values on disks (based on the global step of TF, which will increase every-time we make an optimization step).</p><p>This was only a cursory look at some debugging tools with the eager execution enabled: check out the notebook for a description of other techniques for visualization and debugging.</p><h3>Putting it all together</h3><p>We are finally ready to put it all together! We are going to train our network for a few epochs, using a metric to compute the accuracy and store it in a NumPy array, and a summary to keep track of the loss by saving it on the disk. At the end we use Matplotlib to plot the accuracy across all epochs.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/495f92dc300da5fb54fbbfe5f3458c00/href">https://medium.com/media/495f92dc300da5fb54fbbfe5f3458c00/href</a></iframe><p>The network easily reaches 100% accuracy, this being a simple benchmark:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*iKa1eJxRTSdOgnWquUHSPQ.png" /></figure><p>And if you open up the TensorBoard, you can check the loss’ progress:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*IyozhqraW1-VSCuWen0cSg.png" /></figure><p>You have reached the end of my tutorial on the eager execution. By now, I hope you are convinced that using eager allows for extremely fast prototyping while making the debugging much simpler. Once again, you can check the <a href="https://github.com/tensorflow/workshops/blob/master/extras/eager/eager-tutorial-simone.ipynb">Jupyter notebook</a> if you want more details on all the methods we used in this tutorial, which was only intended as a concise introduction to the interface.</p><p>There are many things we have not seen here, such as how to enable GPU support, or how to use control flow operations inside your model. Nonetheless, I hope this is enough boilerplate code to kick-start you on your journey to the low-level programming beauty of TF!</p><p><em>Originally published at </em><a href="https://medium.com/tensorflow/building-an-iris-classifier-with-eager-execution-13c00a32adb0"><em>medium.com</em></a><em> on April 13, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d77ef2c024" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-developer-experts/building-an-iris-classifier-with-eager-execution-3d77ef2c024">Building an Iris classifier with eager execution</a> was originally published in <a href="https://medium.com/google-developer-experts">Google Developer Experts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an Iris classifier with eager execution]]></title>
            <link>https://medium.com/tensorflow/building-an-iris-classifier-with-eager-execution-13c00a32adb0?source=rss-2e6b05f0ab28------2</link>
            <guid isPermaLink="false">https://medium.com/p/13c00a32adb0</guid>
            <category><![CDATA[community]]></category>
            <dc:creator><![CDATA[Simone Scardapane]]></dc:creator>
            <pubDate>Fri, 13 Apr 2018 18:52:03 GMT</pubDate>
            <atom:updated>2018-04-13T18:52:03.778Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the new additions to TensorFlow in the last months has been the <a href="https://research.googleblog.com/2017/10/eager-execution-imperative-define-by.html">eager execution</a>, an additional low-level interface promising to make development a lot simpler and easier to debug. Whenever eager execution is enabled, operations are executed immediately, instead of having to go through a separate execution step. In a nutshell, this means that writing TF code can be (potentially) <strong>as simple as writing pure NumPy code</strong>!</p><p>Being a long-requested feature, I was looking forward to test it. After playing around for a while with the new interface, I can say the new workflow feels much simpler and intuitive, all the methods are integrated almost perfectly with the rest of TF and, despite being in an experimental stage, most of the functionalities are already stably implemented.</p><p>At the same time, eager execution is a further addition in a framework that is already complex for many newcomers. Which is the reason for this post: a guided tutorial for navigating into this new interface by showing all the steps needed to <strong>build a simple classification model</strong> from scratch on the <a href="https://archive.ics.uci.edu/ml/datasets/iris">Iris dataset</a>. At the end of this article, you will have implemented a fully working model in eager execution from which you can continue experimenting.</p><p>This post is also intended as a <strong>self-contained introduction to the eager execution</strong>: if you are new to low-level programming in TensorFlow, you will learn how you can implement your own algorithms and run them using this new interface. But if you already consider yourself an expert in TF, this post will also highlight the differences when enabling eager execution<em> </em>as opposed to the standard graph construction.</p><p>Everybody is ready? Let’s go!</p><p><em>Note: if you want to experiment with eager execution, I also wrote a </em><a href="https://github.com/tensorflow/workshops/blob/master/extras/eager/eager-tutorial-simone.ipynb"><em>Jupyter notebook</em></a><em> to serve as an extended companion to this post. And if you don’t have a working TensorFlow installation? Remember you can run the notebook using the (completely free) </em><a href="https://colab.research.google.com/"><em>Google Colaboratory service</em></a><em> in the cloud!</em></p><h3>Why do we need eager execution?</h3><p>Nowadays TF includes several high-level APIs, such as <a href="https://www.tensorflow.org/programmers_guide/estimators">Estimators</a> and <a href="https://keras.io/">Keras</a>. But many times, you’d also like to explore and run some low-level code, or dig deeper in the framework. At that point, you may like to write or run something simple to begin with.</p><p>To see how the eager execution works, let’s start by adding and multiplying some numbers, and see how the code would look like <strong>without eager execution</strong>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dda8889daf837256b4395bb10ed7209e/href">https://medium.com/media/dda8889daf837256b4395bb10ed7209e/href</a></iframe><p>If you (like me) have been here since the <a href="http://deeplearning.net/software/theano/">early days of Theano</a>, it is hard to remember just <strong>how strange</strong> this code feels if you are not used to it: why do you even need an external object just to print a number? And why can’t you find the value of b <em>anywhere</em>? The reason is that, internally, TensorFlow is building a computational graph describing all the operations, which is then compiled and optimized behind the scenes — optimization that, in the initial development of TF, was supposed to happen only after the full specification of the model.</p><p>Now compare the same example, but after enabling the eager execution:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ac1d797eef966f0a603e59f00a480b7/href">https://medium.com/media/0ac1d797eef966f0a603e59f00a480b7/href</a></iframe><p>This is pretty much NumPy code! In programming terms, the default API for TF is <strong>declarative</strong>: execution only happens when we request the output of a specific node, and it only returns that particular result. The new eager execution, instead, is <strong>imperative</strong>: execution follows definition immediately, with all of its benefits, immediate visual debugging for one. Let’s see what this means for the optimization of our models — but first, let’s take a small detour to see how you can install and enable eager execution.</p><h3>Enabling eager execution in your code</h3><p>Eager execution has been included as an experimental feature since TF v1.5, and for the purposes of this post I will use the latest v1.7rc0 release (mentioning where you might need to change the code for backward compatibility).</p><p>If you want to play with the latest version, the recommended way is to set up a separate virtual environment (you can find detailed instructions on the notebook and on the <a href="https://github.com/tensorflow/tensorflow">official guide</a>) and install the nightly build of TF from the console:</p><blockquote>pip install tf-nightly[-gpu]</blockquote><p>Eager execution can be enabled with a single line of code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ca97a11d26822fbb84f380d17a79c86/href">https://medium.com/media/0ca97a11d26822fbb84f380d17a79c86/href</a></iframe><blockquote>If you are working with v1.5 or v1.6, change tf.enable_eager_execution() with tfe.enable_eager_execution().</blockquote><p>Importantly, you need to run the command <strong>at the beginning of the program</strong>, otherwise the command will throw an error and refuse to execute. The reason is that, by enabling eager execution, you are changing the default inner mechanisms for TF described before, meaning that several low-level commands will be available only in one of the two modalities (graph construction or eager execution) as described in the documentation.</p><h3>What we will do in this tutorial</h3><p>In the rest of this tutorial, we are going to build a classification model (with eager execution enabled) for the Iris dataset, a simple classification dataset which you should already have encountered when following the <a href="https://www.tensorflow.org/get_started/premade_estimators">starting guide for TF</a>. The task is to classify a set of Irises in three different classes, depending on four numerical features describing their geometrical shape:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xOxhDM-2Cz8RL16rkRBPKQ.jpeg" /><figcaption>The three classes of Irises we want to classify (<a href="https://www.tensorflow.org/get_started/premade_estimators">taken from Getting Started with TensorFlow</a>). From left to right, original copyright by by <a href="https://commons.wikimedia.org/wiki/User:Radomil">Radomil</a>, CC BY-SA 3.0, <a href="https://commons.wikimedia.org/wiki/User:Dlanglois">Dlanglois</a>, CC BY-SA 3.0, and <a href="https://www.flickr.com/photos/33397993@N05">Frank Mayfield</a>, CC BY-SA 2.0.</figcaption></figure><p>For the purpose of this article I am simply loading the dataset’s version from the <a href="http://scikit-learn.org/">scikit-learn library</a>, normalizing its inputs, and splitting it in two for training and testing. If you are curious, here is the full code for loading the dataset:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/66f3c23362e16e555547b4261465e9cc/href">https://medium.com/media/66f3c23362e16e555547b4261465e9cc/href</a></iframe><p>We will proceed in five steps. The first part is just to get some familiarity with the basic eager objects such as variables and functions, and how they interoperate with NumPy arrays. Then, we will learn how to build the classification model (part 2), optimize it (part 3), load the data in TF (part 4), and debug the overall process (part 5).</p><h3>Building the classifier (1/5): functions, gradients, and variables</h3><p>The basic operation provided by eager execution is <strong>automatic differentiation</strong>: given some function operating on tensors, we can compute the gradient with respect to one or more of these tensors automatically (and, it goes without saying, efficiently).</p><p>If you are costructing a TF graph, functions and gradients are defined as nodes inside the computational graph. With the eager execution enabled, instead, <strong>both of them are standard Python functions</strong>. For example, here is some code defining a simple squaring function and computing its gradients:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1dc7b03fa48538213ee6a308b89f689/href">https://medium.com/media/c1dc7b03fa48538213ee6a308b89f689/href</a></iframe><p>Instead of having to build the graph, all operations are internally saved inside a “tape”, transparently to the user. The tfe.gradients_function is where the magic happens: you can pass any Python function (having as input at least one tensor) as argument, and it will return <strong>another Python function</strong> for computing the gradient with respect to any of its arguments.</p><blockquote>You might have noticed that in line 6 above, we use tf.constant to build a constant tensor: this step is optional, since passing a NumPy array or a float value would trigger an automatic casting operation.</blockquote><p>You can also chain the calls to get higher-order derivatives:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ee874e464340d8324881092e4225bd7/href">https://medium.com/media/1ee874e464340d8324881092e4225bd7/href</a></iframe><p>A fundamental building block of TF are <a href="https://www.tensorflow.org/programmers_guide/variables">variables</a>: stateful arrays of numbers that compose our models and that we need to optimize in the training phase. Eager has a custom implementation of variables:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7c6a0ff055341b0550b40b8b1490dac/href">https://medium.com/media/e7c6a0ff055341b0550b40b8b1490dac/href</a></iframe><p>As you can see, variables in eager are <strong>fully interoperable with NumPy arrays</strong>, and their value is automatically initialized as soon as they are requested (while with graph construction you would need to explicitly call an initialization routine). Using tfe.implicit_gradients , you can also automatically compute the gradients of a function with respect to <em>all</em> variables it uses. For example, this is equivalent to the gradient computation above, but using variables and implicit gradients:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2bad62fc732b83ad9978bcb838c16b3c/href">https://medium.com/media/2bad62fc732b83ad9978bcb838c16b3c/href</a></iframe><p>Now that we have all the building blocks in hand, we can move to a higher level of abstraction, starting with the definition of our neural network.</p><h3>Building the classifier (2/5): layers and networks</h3><p>In theory, we already have everything we need for optimizing a model: we can instantiate a bunch of variables (the parameters of our models), and encapsulate the logic of the model itself inside a function. For example, this is a one-dimensional logistic regression model:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e221076be0b0a4dda7399c271e1dc0cc/href">https://medium.com/media/e221076be0b0a4dda7399c271e1dc0cc/href</a></iframe><p>However, as soon as we start adding more operations (e.g., new hidden layers, dropout, …) this approach does not scale. A simple way to build more complicated models is to use <a href="https://www.tensorflow.org/tutorials/layers">layers</a>, which should be quite familiar to anyone using the high-level interface of TF:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3db2a636548ca145c291279b241e57b8/href">https://medium.com/media/3db2a636548ca145c291279b241e57b8/href</a></iframe><p>In the above example, we are building a model with a single hidden layer and a dropout operation in the middle: the <strong>definition of variables is taken care of inside each layer</strong>, and we can define our model by chaining together the layers inside a simple (once more, Python) function. The model is quite simple to interpret, even if you are not used to TF, and we can implement the dropout logic by just passing another Boolean flag to the function, adding to the readability.</p><blockquote>The tf.layers module is an implementation of tf.keras.layers , which contains a set of high-level layers defined inside the <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers">Keras library</a>. The implementation at this moment is only partial, with new layers being added continuously: in the meantime, you can also use directly the layers defined in tf.keras.layers .</blockquote><p>Since building networks in this fashion is a common workflow, TF also provides a further level of abstraction with the Model object, a custom class that encapsulates the model’s logic, providing a few advanced tools for debugging and checkpointing the training:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98e4757f0e54d53fda1538558b13eca2/href">https://medium.com/media/98e4757f0e54d53fda1538558b13eca2/href</a></iframe><blockquote>If you are working with v1.5 or v1.6, you need to use tfe.Network instead of tf.keras.Model , and you need to explicitly call self.track_layer() for each layer added in the initialization.</blockquote><p>A major benefit of networks is that they encapsulate all the logic concerning the variables of the model. To understand it, let’s play around with our model a little bit:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b5298c8c6fe542c78e6f002535f2329c/href">https://medium.com/media/b5298c8c6fe542c78e6f002535f2329c/href</a></iframe><p>Line 2 is just the initialization of the model. The fifth line shows the <strong>lazy nature </strong>of variables inside a model: even though eager execution has a dynamic workflow, variables are only initialized when a model is run for the first time, because prior to that eager has no indication on the shape of its input tensors. Predictions can be obtained by simply calling the object with some data (line 8): after this, variables are correctly initialized, as shown in line 11.</p><h3>Building the classifier (3/5): losses and optimizers</h3><p>Since we are going to use our model to do some classification, we need to define a proper cost function to train it. Similarly to before, we can use the standard components of TF, by <strong>encapsulating them in a Python function</strong>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d63945fea97386be94f4cb653989a847/href">https://medium.com/media/d63945fea97386be94f4cb653989a847/href</a></iframe><blockquote>As a quick sidenote, we are using softmax_cross_entropy_with_logits_v2 , as the old softmax_cross_entropy_with_logits will get deprecated soon! The two are entirely identical here, the difference being that the new version allows for gradient computation with respect to the labels (not needed for this simple example). By removing the one-hot encoding, we can also use sparse_softmax_cross_entropy equivalently.</blockquote><p>For the optimization, we can define our custom logic or (more probably) use one of the <a href="https://www.tensorflow.org/api_guides/python/train">many optimizers already defined inside TF</a>. At this point, you will probably guess what our code will look like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e81e13096d7e2c95301cf58aaddbbd93/href">https://medium.com/media/e81e13096d7e2c95301cf58aaddbbd93/href</a></iframe><p>The only syntactic sugar here is using an anonymous function with no arguments to call the minimization routine: this is needed because the optimizers were designed to work for the graph construction, and have been customized to also support the new eager interface. Another possibility is to explicitly compute the gradients of our model, and use the apply_gradients function of the optimizer: if you are interested I describe this alternative way in the notebook.</p><h3>Building the classifier (4/5): loading data and iterators</h3><p>If you have kept up-to-date with TF, you will know that the preferred way to feed data to our models is though the <a href="https://www.tensorflow.org/programmers_guide/datasets">high-level Dataset API</a>, which can be used to load data from multiple sources and connects with all the high-level APIs. A nice feature of eager execution is that you can encapsulate datasets inside a Pythonesque iterator, and use the iterator to <strong>cycle over it as you would any other iterator</strong>.</p><p>Let’s loop over the training portion of the Iris dataset we loaded before using an iterator:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c1afd1ef27ff16f37b57261c4e4d1e7/href">https://medium.com/media/6c1afd1ef27ff16f37b57261c4e4d1e7/href</a></iframe><p>In the above code we are loading the data inside the Dataset object (line 2), and then using the iterator to loop over mini-batches of 32 elements, computing the average number of positive labels for each batch, e.g.:</p><blockquote>Percentage of class [0]: 31.25 % <br>Percentage of class [0]: 34.375 % <br>Percentage of class [0]: 37.5 % <br>Percentage of class [0]: 25.0 %</blockquote><h3>Building the classifier (5/5): Debugging tools</h3><p>The last thing we need for our task are some tools for debugging and visualizing the results. Because in the standard graph construction the execution is masked underneath the session’s dynamics, debugging has always been a difficult point for newcomers to TF, and the main reason why the <a href="https://www.tensorflow.org/programmers_guide/summaries_and_tensorboard">TensorBoard</a> was developed. In eager execution, however, everything runs dynamically and you can use <strong>any tool in your Python’s toolbox</strong> for it: from Matplotlib’s plots to the console’s outputs, you name it.</p><p>Eager has also a few utilities to simplify the debugging. For example, <a href="https://www.tensorflow.org/versions/master/api_docs/python/tf/contrib/eager/metrics">metrics</a> can be used to compute and accumulate values across each epoch:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef3f0403891bea44f5db1e518630f72f/href">https://medium.com/media/ef3f0403891bea44f5db1e518630f72f/href</a></iframe><p>The code above loops through a single epoch, and for each batch it computes the accuracy of the model. As we have not trained it yet, the accuracy will probably hoover around 33%, or random chance.</p><p>And if you are fond of the good ol’ TensorBoard, you can save values for visualization on the dashboard using the <a href="https://www.tensorflow.org/versions/master/api_docs/python/tf/contrib/summary">experimental implementation of the summaries</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/58af014c22dd1a8202f311589da92417/href">https://medium.com/media/58af014c22dd1a8202f311589da92417/href</a></iframe><p>The syntax here is slightly more involved: after creating a writer, we need to set it as the default one <em>and</em> choose an interval for saving the values on disks (based on the global step of TF, which will increase every-time we make an optimization step).</p><p>This was only a cursory look at some debugging tools with the eager execution enabled: check out the notebook for a description of other techniques for visualization and debugging.</p><h3>Putting it all together</h3><p>We are finally ready to put it all together! We are going to train our network for a few epochs, using a metric to compute the accuracy and store it in a NumPy array, and a summary to keep track of the loss by saving it on the disk. At the end we use Matplotlib to plot the accuracy across all epochs.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/495f92dc300da5fb54fbbfe5f3458c00/href">https://medium.com/media/495f92dc300da5fb54fbbfe5f3458c00/href</a></iframe><p>The network easily reaches 100% accuracy, this being a simple benchmark:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*iKa1eJxRTSdOgnWquUHSPQ.png" /></figure><p>And if you open up the TensorBoard, you can check the loss’ progress:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*IyozhqraW1-VSCuWen0cSg.png" /></figure><h3>A few final words</h3><p>You have reached the end of my tutorial on the eager execution. By now, I hope you are convinced that using eager allows for extremely fast prototyping while making the debugging much simpler. Once again, you can check the <a href="https://github.com/tensorflow/workshops/blob/master/extras/eager/eager-tutorial-simone.ipynb">Jupyter notebook</a> if you want more details on all the methods we used in this tutorial, which was only intended as a concise introduction to the interface.</p><p>There are many things we have not seen here, such as how to enable GPU support, or how to use control flow operations inside your model. Nonetheless, I hope this is enough boilerplate code to kick-start you on your journey to the low-level programming beauty of TF!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13c00a32adb0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tensorflow/building-an-iris-classifier-with-eager-execution-13c00a32adb0">Building an Iris classifier with eager execution</a> was originally published in <a href="https://medium.com/tensorflow">TensorFlow</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>