<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by dan lee on Medium]]></title>
        <description><![CDATA[Stories by dan lee on Medium]]></description>
        <link>https://medium.com/@Adline125?source=rss-e25490321c4b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*JojwbHFRf8UP2HGK.jpg</url>
            <title>Stories by dan lee on Medium</title>
            <link>https://medium.com/@Adline125?source=rss-e25490321c4b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 13:59:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Adline125/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Understanding TensorFlow: Part 4]]></title>
            <link>https://medium.com/@Adline125/understanding-tensorflow-94bdea8e1fd9?source=rss-e25490321c4b------2</link>
            <guid isPermaLink="false">https://medium.com/p/94bdea8e1fd9</guid>
            <dc:creator><![CDATA[dan lee]]></dc:creator>
            <pubDate>Thu, 09 Sep 2021 14:14:48 GMT</pubDate>
            <atom:updated>2021-10-31T05:27:52.128Z</atom:updated>
            <content:encoded><![CDATA[<h4>Serie 4: Implementing our first neural network</h4><p>Today’s outline:</p><ol><li>Implementing our first neural network</li><li>Preparing the data</li><li>Buiding neural network model</li><li>Defining loss and optimizer</li><li>Training the neural network</li></ol><h4>Implementing our first neural network</h4><p>Great! Now that you’ve learned the most basic operations of TensorFlow 2.x, it’s high time that we move on and implement something moderately complex. Let’s implement a neural network. Precisely, we will implement a fully connected neural network model.</p><p>One of the stepping stones to the introduction of neural networks is to implement a neural network that is able to classify digits. For this task, we will be using the famous MNIST dataset made available at <a href="http://yann.lecun.com/exdb/mnist/">http://yann.lecun.com/exdb/mnist/.</a></p><p>You might feel a bit skeptical regarding our using a computer vision task rather than an NLP task. However, vision tasks can be implemented with less preprocessing and are easy to understand.</p><p>As this is our first encounter with neural networks, we will walk through the main parts of the example. However, note that I will only walk through the crucial bits of the exercise.</p><h3>Preparing the data</h3><p>In TensorFlow 2.x, the model inputs are Tensors with concrete value. As long as we download the dataset mnist and preprocess it to a Tensor with normalization and batch, the model inputs would be ready.</p><p>The mnist data set is available at this link <a href="http://yann.lecun.com/exdb/mnist/">http://yann.lecun.com/exdb/mnist/</a>. Download it and we will read it directly. Don’t forget to change the data path to where you put the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VpfriiCakyrE5D4PWYK3pA.png" /></figure><p>Now, we can process the data one more step with tf.data.Dataset. The iteration mechanism of it happens in a streaming fashion, so the full dataset does not need to fit into memory. And it is convenient to bach the dataset with tf.data.Dataset.from_tensor_slices().batch(). from_tensor_slices() will slice you dataset (a Python list) into item by item. batch() will divide them into batch number groups.</p><p>train_ds = tf.data.Dataset.from_tensor_slices(</p><p>(x_train, y_train)).shuffle(10000).batch(32)</p><p>test_ds = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(32)</p><h3>Buiding neural network model</h3><p>There are various ways to build a neural network model. It can be quite complicated using lower-level API to build every block you need by yourself or can be quite simple by using higher-level API to quickly check and run a model to evaluate your task solution. In practice, the task-oriented situation is much more common. tf.keras API would be a better choice to find the balance between general use and customization.</p><p>If you are a beginner of TensorFlow and want to witness the magic of a neural network as soon as possible. tf.keras.Sequential model would be a nice choice. It is the easiest and clearest way to build your model. Two steps are needed:</p><ul><li>Think of the layers and their order that you want to use in your model.</li><li>List the above layers in a list and send them to tf.keras.Sequential as its parameter</li></ul><p>In our example, We want to use two full connection layers, tf.keras.layers.Dense, to finish our task. But tf.keras.layers.Dense only accept a Tensor with the shape of 1 axis. The minst data are images with shape [28, 28]. There are 2 axes. So we need a layer tf.keras.layers.Flatten to change data shape to [28 * 28] with 1 axis. Now we solved our shape incompatible problem. Next, full connection layer has a problem. There are too many parameters in it which leads to overfitting easily. tf.keras.layers.Dropout can get rid of a certain percentage of parameters randomly to prevent overfitting. e.g. 20% parameters would be dumped if you pass 0.2 to it.</p><p>Now we know what layers we may use in our model. Let’s put them in order to a list, then send the list to our tf.keras.Sequential model.</p><p>model = tf.keras.models.Sequential([</p><p>tf.keras.layers.Flatten(input_shape=(28, 28)),</p><p>tf.keras.layers.Dense(128, activation=’relu’),</p><p>tf.keras.layers.Dropout(0.2),</p><p>tf.keras.layers.Dense(10)</p><p>])</p><p>Another way to build a neural network model with tf.kears API is model subclassing API. Define the model as a class that takes heritage from tf.keras.Model. There are two necessary functions to be defined in your class. the __init__() and the call(). In the __init__() function, the two steps to building your model we mentioned above can apply here with a little change.</p><ul><li>Think of the layers and their order that you want to use in your model.</li><li>List the above layers one by one in order as the class member of your model.</li></ul><p>This time we want to add a new layer called CNN to our model. CNN usually results in better performance. tf.keras.layers.Conv2D is a kind of CNN layer that accepts Tensor shape with [height, width, channels] or [channels, height, width]. e.g. input_shape=(128, 128, 3) for 128x128 RGB pictures in data_format=”channels_last”.</p><p>In our example, the CNN layer can accept our image data shape naturally. CNN layer would be the first layer to accept the inputs directly in our model and output a certain number (specified by layer parameter ‘filters’) of Tensors with the shape of [height, width] or a similar shape influenced by parameter `padding`. We can ignore the difference when `padding` is set to be different value by now. What we need to know is that both situations will produce a Tensor with the shape that needs to be flattened if the CNN layer wants to be followed by a full connection layer.</p><p>The call() function defines the operations between these layers we defined above. We send input data to the CNN layers and send the result to the next layer and so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WUYT8j2Uuw2RKHfZmrXPow.png" /></figure><h3>Defining loss and optimizer</h3><p>No matter which method you use to build your model, loss function and optimizer must be specified before you start to train your model.</p><p>In our example, our task is a 10-class classification which is a sparse category task. We will use SparseCategoricalCrossentropy to be our model loss. SparseCategoricalCrossentropy computes the sparse categorical cross-entropy loss.</p><p>The optimizer will be set to be Adam. The optimization problem has a developing history from SGD, Momentum, Nesterov Accelerated Gradient, AdaGrad, RMSProp to Adam. They influence and fix problems of one to another. Since the optimization problem is not the focus of this chapter. let’s just remember the conclusion that you should use Adam in most cases.</p><p>loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)</p><p>optimizer = tf.keras.optimizers.Adam()</p><h3>Training the neural network</h3><p>Here we will list the training code only. You can find the complete code of both training and testing in the <em>tensorflow_introduction.ipynb</em> file in the ch2 folder.</p><p>With tf.keras.sequence model, we train our model only with two lines.</p><p>model.compile(optimizer=optimizer,</p><p>loss=loss,</p><p>metrics=[‘accuracy’])</p><p>model.fit(x_train, y_train, epochs=5)</p><p>If you want to define the training process by yourself, we need GradientTape which we introduced in the last section. GradientTape will help us to calculate the gradient with respect to loss and model parameters for each step. Then the optimizer will apply the gradient to update model parameters.</p><p>For each step, there should be a metric to tell us the performance the model is getting better or not. The metric which is commonly chosen is Accuracy.</p><p>First, define a function of what to do in each step. we will list the operations we mentioned above in this function including calculating loss, calculating gradients, apply gradients and calculating accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SJUBfb8l7ap3aC5_DemDnQ.png" /></figure><p>Second, start to train. Specify how many epochs you want to train. Then input your data to the train step function until the epochs is finish iterating.</p><p>In this code, accuracy is a function that takes some predictions and labels as inputs and provides the accuracy (how many predictions matched the actual label). It is defined in the exercise file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UvsUZDt1_D-dF2PVWsXimQ.png" /></figure><p>If successful, you should be able to see behavior similar to the ones shown in Figure 2.10. After 50 epochs, the test accuracy should reach approximately 98%:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*UeXk7t4lsQjSuI56YZY7-g.png" /><figcaption>Figure 2.10: Training loss and test accuracy for the MNIST digit classification task</figcaption></figure><h3>Summary</h3><p>In this series, you took your first steps to solve NLP tasks by understanding the difference between TensorFlow 1.x and TensorFlow 2.x on which we will be implementing our algorithms. First, we discussed the eager execution which enables us to program in a much more intuitive way. Next, we discussed autograph (tf.function) which supports automatically construct graphs when you need them.</p><p>Then we discussed basic operations of TensorFlow 2.x including Tensor creation, merge and split, Tensor comparison, mathematical operations, and neural network-related operations. Later, we brought all these elements together to implement a neural network to classify an MNIST dataset.</p><p><strong>Other series of Understanding TensorFlow:</strong></p><p>Serie1: <a href="https://medium.com/@Adline125/understanding-tensorflow-series-979e71cc5562">https://medium.com/@Adline125/understanding-tensorflow-series-979e71cc5562</a></p><p>Serie2: <a href="https://medium.com/@Adline125/understanding-tensorflow-fcc431891d08">https://medium.com/@Adline125/understanding-tensorflow-fcc431891d08</a></p><p>Serie3–1: <a href="https://medium.com/@Adline125/understanding-tensorflow-ce18f0e1bbbc">https://medium.com/@Adline125/understanding-tensorflow-ce18f0e1bbbc</a></p><p>Serie3–2: <a href="https://medium.com/@Adline125/understanding-tensorflow-2c6496b71368">https://medium.com/@Adline125/understanding-tensorflow-2c6496b71368</a></p><p>Serie4: <a href="https://medium.com/@Adline125/understanding-tensorflow-94bdea8e1fd9">https://medium.com/@Adline125/understanding-tensorflow-94bdea8e1fd9</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94bdea8e1fd9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding TensorFlow: Part 3–2]]></title>
            <link>https://medium.com/@Adline125/understanding-tensorflow-2c6496b71368?source=rss-e25490321c4b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c6496b71368</guid>
            <dc:creator><![CDATA[dan lee]]></dc:creator>
            <pubDate>Thu, 09 Sep 2021 13:54:47 GMT</pubDate>
            <atom:updated>2021-10-31T05:27:20.977Z</atom:updated>
            <content:encoded><![CDATA[<h4>Serie 3–2: Neural network-related operations</h4><p>Today’s outline:</p><ol><li>Defining loss</li><li>Automatic differentiation and gradients</li><li>Optimization of neural networks</li></ol><h3>Defining loss</h3><p>We know that in order for a neural network to learn something useful, a loss needs to be defined. There are several functions for automatically calculating the loss in TensorFlow, two of which are shown in the following code. The tf.nn.l2_loss function is the mean squared error loss, and tf.nn.softmax_cross_entropy_ with_logits_v2 is another type of loss, which actually gives better performance in classification tasks. And by logits here, we mean the unnormalized output of the neural network (that is, the linear output of the last layer of the neural network):</p><p># Returns half of L2 norm of t given by sum(t**2)/2</p><p>x = tf.constant([[2,4],[6,8]],dtype=tf.float32)</p><p>x_hat = tf.constant([[1,2],[3,4]],dtype=tf.float32)</p><p># MSE = (1**2 + 2**2 + 3**2 + 4**2)/2 = 15</p><p>MSE = tf.nn.l2_loss(x-x_hat)</p><p># A common loss function used in neural networks to optimize the network</p><p># Calculating the cross_entropy with logits (unnormalized outputs of the last layer)</p><p># instead of outputs leads to better numerical stabilities</p><p>y = tf.constant([[1,0],[0,1]],dtype=tf.float32)</p><p>y_hat = tf.constant([[3,1],[2,5]],dtype=tf.float32)</p><p># This function alone doesn’t average the cross entropy losses of all data points,</p><p># You need to do that manually using reduce_mean function</p><p>CE = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=y_hat))</p><p>tf.keras has the corresponding operation tf.keras.MSE and tf.keras.losses.categorical_crossentropy. Note that tf.nn.l2_loss automatically compute sum(t**2)/2 while tf.keras.MSE need to plus sum operation manually by tf.reduce_sum. tf.keras.losses.categorical_crossentropy needs to specify parameter from_logits = True to take the unnormalized output of the neural network to compute.</p><p>#MSE_keras</p><p>x = tf.constant([[2,4],[6,8]],dtype=tf.float32)</p><p>x_hat = tf.constant([[1,2],[3,4]],dtype=tf.float32)</p><p>MSE_keras = tf.reduce_sum(tf.keras.losses.MSE(x, x_hat))</p><p># cross entropy</p><p>y = tf.constant([[1,0],[0,1]],dtype=tf.float32)</p><p>y_hat = tf.constant([[3,1],[2,5]],dtype=tf.float32)</p><p>CE_keras = tf.reduce_mean(tf.keras.losses.categorical_crossentropy(y, y_hat, from_logits=True))</p><h3>Automatic differentiation and gradients</h3><h4>tf.Variable</h4><p>Since in most cases, you will want to calculate gradients with respect to a model’s trainable variables. Here before we introduce how to use TensorFlow to calculate gradients, we need to introduce tf.Variable first.</p><p>Variables play an important role in TensorFlow. A variable is essentially a tensor with a specific shape defining how many dimensions the variable will have and the size of each dimension. However, unlike a regular tensor, variables are mutable; meaning that the value of the variables can change after they are defined. This is an ideal property to have to implement parameters of a learning model (for example, neural network weights), where the weights change slightly after each step of learning. For example, if you define a variable with x = tf.Variable(0,dtype=tf.int32), you can change the value of that variable using a TensorFlow operation such as tf.assign(x,x+1). However, if you define a tensor such as x = tf.constant(0,dtype=tf.int32), you cannot change the value of the tensor, as for a variable. It should stay 0 until the end of the program execution.</p><p>Variable creation is quite simple. When creating a variable, a few things are of high importance. We list them here and discuss each in detail in the following paragraphs:</p><ul><li>Variable shape</li><li>Data type</li><li>initial value</li><li>Name (optional)</li></ul><p>The variable shape is a 1D vector of the [x,y,z,…] format. Each value in the list indicates how large the corresponding dimension or axis is. For instance, if you require a 2D tensor with 50 rows and 10 columns as the variable, the shape would be equal to [50,10].</p><p>When creating a variable, a few things are of high importance. We list them here and discuss each in detail in the following paragraphs:</p><ul><li>Variable shape</li><li>Data type</li><li>Initial value</li><li>Name (optional)</li></ul><p>The variable shape is a 1D vector of the [x,y,z,…] format. Each value in the list indicates how large the corresponding dimension or axis is. For instance, if you require a 2D tensor with 50 rows and 10 columns as the variable, the shape would be equal to [50,10].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9CER5OcAqFlqz-7oDjFb0g.png" /></figure><p>The data type plays an important role in determining the size of a variable. There are many different data types including the commonly used tf.bool, tf.uint8, tf.float32, and tf.int32. Each data type has a number of bits required to represent a single value with that type. For example, tf.uint8 requires 8 bits, whereas tf.float32 requires 32 bits. It is common practice to use the same data types for computations as doing otherwise can lead to data type mismatches. So if you have two different data types for two tensors that you need to transform, you have to explicitly convert one tensor to the other tensor’s type using the tf.cast(…) operation.</p><p>The tf.cast(…) operation is designed to cope with such situations. For example, if you have an x variable with the tf.int32 type, which needs to be converted to tf.float32, employ tf.cast(x,dtype=tf.float32) to convert x to tf.float32.</p><p>Next, a variable requires an <em>initial </em>value to be initialized with. TensorFlow provides several different initializers for our convenience, including constant initializers and normal distribution initializers. Here are a few popular TensorFlow initializers you can use to initialize variables:</p><ul><li>tf.zeros</li><li>tf.constant</li><li>tf.random_uniform</li><li>tf.truncated_normal</li><li>tf.normal</li></ul><p>Finally, the <em>name </em>of the variable will be used as an ID to identify that variable in the graph. So if you ever visualize the computational graph, the variable will appear by the argument passed to the name keyword. If you do not specify a name, TensorFlow will use the default naming scheme.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wWbyVYxHsh5LZWNfIu3GPw.png" /></figure><p>We mentioned at the very beginning of this section that the data type in TensorFlow is Tensor. So tf.Variable is a Tensor, too. But it is a little bit special comparing with other common Tensors. tf.Variable objects store mutable tf.Tensor. More accurately speaking, variables are backed by tensor. they have a dtype and shape, and can be exported to NumPy just like other Tensors. The difference between tf.Variable and other Tensors is that variables cannot be reshaped and common Tensors can’t be reassigned by using <a href="#assign">tf.Variable.assign</a>.</p><p># create variable</p><p>a = tf.Variable([2.0, 3.0])</p><p># reassign</p><p>a.assign([1, 2])</p><p>The parameter ‘trainable’ in tf.Variable needs to be noted. If a tf.Variable is set by trainable=False, the gradient computation with respect to it would be ignored. The default value of ‘trainable’ is True.</p><p>One more thing, tf.Variable + tf.Tensor would be a tf.Tensor.</p><p>a = tf.Variable(1.0)</p><p>b = tf.constant(2.0)</p><p>c = a + b</p><p>print(c)</p><p># Returns (out) =&gt;</p><p>tf.Tensor(3.0, shape=(), dtype=float32)</p><h4>tf.GradientTape</h4><p>TensorFlow2.x provides the tf.GradientTape API for automatic differentiation. It ‘records’ operations happened during the forward pass to a ‘tape’, then uses that tape in reverse order to compute the gradients during the backward pass.</p><p>Let’s see an example to calculate gradients with respect to a model’s trainable variables.</p><p># trainable variables in a model is watched by default</p><p>layer = tf.keras.layers.Dense(2, activation=’relu’)</p><p>x = tf.constant([[1., 2., 3.]])</p><p>with tf.GradientTape() as tape:</p><p># Forward pass</p><p>y = layer(x)</p><p>loss = tf.reduce_mean(y**2)</p><p># Calculate gradients with respect to every trainable variable</p><p>grad = tape.gradient(loss, layer.trainable_variables)</p><p># Returns (out) =&gt;</p><p>[&lt;tf.Tensor: shape=(3, 2), dtype=float32, numpy= array([[0. , 1.0097853], [0. , 2.0195706], [0. , 3.029356 ]], dtype=float32)&gt;,</p><p>&lt;tf.Tensor: shape=(2,), dtype=float32, numpy=array([0. , 1.0097853], dtype=float32)&gt;]</p><p><a href="https://www.tensorflow.org/api_docs/python/tf/GradientTape">tf.GradientTape</a> automatically records (watch) operations after accessing a trainable tf.Variable. This default behavior makes it convenient to calculate the gradient of a loss with respect to all a model’s trainable variables. But <a href="https://www.tensorflow.org/api_docs/python/tf/Tensor">tf.Tensor</a> is not “watched” by default. So TensorFlow provides us a <a href="#watch">GradientTape.watch</a>() method to have over the control of whose gradient should be calculated.</p><p># use <a href="#watch">GradientTape.watch</a>() to take over the control</p><p>x0 = tf.Variable(0.0, trainable=False)</p><p>x1 = tf.Variable(10.0)</p><p>x2 = tf.constant(4.0)</p><p>with tf.GradientTape() as tape:</p><p>tape.watch(x2)</p><p>y0 = tf.math.sin(x0)</p><p>y1 = tf.nn.softplus(x1)</p><p>y2 = tf.pow(x2, 2.0)</p><p>y = y0 + y1 + y2</p><p>y_sum = tf.reduce_sum(y)</p><p>grad = tape.gradient(y_sum, {‘x0’: x0, ‘x1’: x1, ‘x2’: x2})</p><p>print(‘dy/dx0:’, grad[‘x0’])</p><p>print(‘dy/dx1:’, grad[‘x1’].numpy())</p><p>print(‘dy/dx2:’, grad[‘x2’])</p><p># Returns (out) =&gt;</p><p>dy/dx0: None</p><p>dy/dx1: 0.9999546</p><p>dy/dx2: tf.Tensor(8.0, shape=(), dtype=float32)</p><p>we can see from the above example, x0 is not trainable and not watched, the GradientTape ignored it. The gradient result with respect to x0 is None. x1 is a trainable tf.Variable. It is watched by GradientTape automatically and it has a gradient result. x3 is a constant. It needs to be explicitly watched by GradientTape so that it can get a gradient result.</p><h3>Optimization of neural networks</h3><p>After defining the loss of a neural network, our objective is to minimize that loss over time. Optimization is the procedure used for this. In other words, the objective of the optimizer is to find the neural network parameters (that is, weights and bias values) that give the minimum loss for all the inputs. Again, our beloved TensorFlow provides us with several different optimizers, so we don’t have to worry about implementing them from scratch.</p><p><em>Figure 2.9 </em>illustrates a simple optimization problem and shows how the optimization happens over time. The curve can be imagined as the <em>loss curve </em>(for high dimensions, we say <em>loss surface</em>), where <em>x </em>can be thought of as the parameters of the neural network (in this case a neural network with a single weight), and <em>y </em>can be thought of as the loss. We have an initial guess of <em>x=2</em>. From this point, we use the optimizer to reach the minimum <em>y </em>(that is, loss), which is obtained at <em>x=0</em>. More specifically, we take small steps in the direction opposite to the gradient at a given point and continue for several steps in this manner. However, in real-world problems, the loss surface will not be as nice as in the illustration, but it will be more complex:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*mBauEqw9VGLGcUMcs1mCFw.png" /><figcaption>Figure 2.9: The optimization process</figcaption></figure><p>In this example, we use tf.keras.optimizers.SGD. The learning_rate parameter denotes the step size you take in the direction of minimization (distance between two red dots):</p><p># Optimizers play the role of tuning neural network parameters so that</p><p># their task error is minimal</p><p># For example task error can be the MSE</p><p># for a classification task</p><p>opt = tf.keras.optimizers.SGD(learning_rate=0.1)</p><p>var = tf.Variable(1.0)</p><p>loss = lambda: (var ** 2)/2.0 # d(loss)/d(var1) = var1</p><p># First step is `- learning_rate * grad`</p><p>opt.minimize(loss, [var]).numpy()</p><p>print(var.numpy())</p><p># Returns (out) =&gt;0.9</p><p>Everytime you execute the loss minimize operation , you will get close to the varvalue that gives the minimum of loss.</p><p>If you want to use GradientTape to calculate the gradients by yourself, you can use optimizer.apply_gradients to optimize loss. Here is the example:</p><p>var = tf.Variable(1.0)</p><p>optimizer = tf.keras.optimizers.SGD(learning_rate=0.1)</p><p>with tf.GradientTape() as tape:</p><p># loss</p><p>loss = (var ** 2)/2.0</p><p>grads = tape.gradient(loss, [var])</p><p>optimizer.apply_gradients(zip(grads, [var]))</p><p>print(var.numpy())</p><p># Returns (out) =&gt;0.9</p><p><strong>Other Series of Understanding TensorFlow:</strong></p><p>Serie1: <a href="https://medium.com/@Adline125/understanding-tensorflow-series-979e71cc5562">https://medium.com/@Adline125/understanding-tensorflow-series-979e71cc5562</a></p><p>Serie2: <a href="https://medium.com/@Adline125/understanding-tensorflow-fcc431891d08">https://medium.com/@Adline125/understanding-tensorflow-fcc431891d08</a></p><p>Serie3–1: <a href="https://medium.com/@Adline125/understanding-tensorflow-ce18f0e1bbbc">https://medium.com/@Adline125/understanding-tensorflow-ce18f0e1bbbc</a></p><p>Serie3–2: <a href="https://medium.com/@Adline125/understanding-tensorflow-2c6496b71368">https://medium.com/@Adline125/understanding-tensorflow-2c6496b71368</a></p><p>Serie4: <a href="https://medium.com/@Adline125/understanding-tensorflow-94bdea8e1fd9">https://medium.com/@Adline125/understanding-tensorflow-94bdea8e1fd9</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c6496b71368" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding TensorFlow: Part 3–1]]></title>
            <link>https://medium.com/@Adline125/understanding-tensorflow-ce18f0e1bbbc?source=rss-e25490321c4b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce18f0e1bbbc</guid>
            <dc:creator><![CDATA[dan lee]]></dc:creator>
            <pubDate>Thu, 09 Sep 2021 13:33:58 GMT</pubDate>
            <atom:updated>2021-10-31T05:26:11.001Z</atom:updated>
            <content:encoded><![CDATA[<h4>Serie 3–1: Neural network-related operations</h4><p>Today’s outline:</p><ol><li>Introduction</li><li>Nonlinear activations used by neural networks</li><li>The convolution operation</li><li>The pooling operation</li></ol><h3>1. Introduction</h3><p>Now let’s look at several useful neural network-related operations that we will use heavily in the following chapters. The operations we will discuss here range from simple element-wise transformations (that is, activations) to computing partial derivatives of a set of parameters with respect to another value. We will also implement a simple neural network as an exercise.</p><p>TensorFlow 2.x has incorporated Keras into its own system, known as tf.keras Some operations would have two ways to be done. In practice, tf.keras is quite convenient to use. So we will introduce both tf.keras and tf for each operation.</p><h3>2. Nonlinear activations used by neural networks</h3><p>Nonlinear activations enable neural networks to perform well at numerous tasks. Typically, there is a nonlinear activation transformation (that is, activation layer) after each layer output in a neural network (except for the last layer). A nonlinear transformation helps a neural network to learn various nonlinear patterns that are present in data. This is very useful for complex real-world problems, where data often has more complex nonlinear patterns, in contrast to linear patterns. If not for the nonlinear activations between layers, a deep neural network will be a bunch of linear layers stacked on top of each other. Also, a set of linear layers can essentially be compressed to a single bigger linear layer. In conclusion, if not for the nonlinear activations, we cannot create a neural network with more than one layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Z4S6dGxpldSKVKTq_0B8w.png" /></figure><p>Now we’ll list two commonly used nonlinear activations in neural networks and how they can be implemented in TensorFlow. :</p><p># Sigmoid activation of x is given by 1 / (1 + exp(-x))</p><p>tf.nn.sigmoid(x,name=None)</p><p>or</p><p>tf.keras.activations.sigmoid(x)</p><p># ReLU activation of x is given by max(0,x)</p><p>tf.nn.relu(x, name=None)</p><p>or</p><p>tf.keras.activations.relu(x, alpha=0.0, max_value=None, threshold=0)</p><h3>3. The convolution operation</h3><p>A convolution operation is a widely used signal-processing technique. For images, convolution is used to produce different effects of an image. An example of edge detection using convolution is shown in <em>Figure 2.6</em>. This is achieved by shifting a convolution filter on top of an image to produce a different output at each location (see <em>Figure 2.7 </em>later in this section). Specifically, at each location we do element-wise multiplication of the elements in the convolution filter with the image patch (same size as the convolution filter) that overlaps with the convolution filter and takes the sum of the multiplication:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*qDBM0bdYDHUYpgWa3fUabQ.png" /><figcaption>Figure 2.6: Using the convolution operation for edge detection in an image(Source: <a href="https://en.wikipedia.org/wiki/Kernel_(image_processing))">https://en.wikipedia.org/wiki/Kernel_(image_processing))</a></figcaption></figure><p>The following is the implementation of the convolution operation:</p><p>x = tf.constant( [[</p><p>[[1],[2],[3],[4]],</p><p>[[4],[3],[2],[1]],</p><p>[[5],[6],[7],[8]],</p><p>[[8],[7],[6],[5]]</p><p>]],</p><p>dtype=tf.float32)</p><p>x_filter = tf.constant( [</p><p>[</p><p>[[0.5]],[[1]]</p><p>], [</p><p>[[0.5]],[[1]]</p><p>]</p><p>],</p><p>dtype=tf.float32)</p><p>x_stride = [1,1,1,1]</p><p>x_padding = ‘VALID’</p><p>x_conv = tf.nn.conv2d( input=x, filters=x_filter, strides=x_stride, padding=x_padding)</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[[[ 7.5]</p><p>[ 7.5]</p><p>[ 7.5]]</p><p>[[13.5]</p><p>[13.5]</p><p>[13.5]]</p><p>[[19.5]</p><p>[19.5]</p><p>[19.5]]]], shape=(1, 3, 3, 1), dtype=float32)</p><p>Here, the apparently excessive number of square brackets used might make you think that the example can be made easy to follow by getting rid of these redundant brackets. Unfortunately, that is not the case. For the tf.nn.conv2d(…) or tf.keras.layers.conv2d(…) operation, TensorFlow requires input, filter, and stride to be of an exact format. We will now go through each argument in conv2d(input, filter, strides, padding) in more detail:</p><p><strong>input</strong>: This is typically a 4D tensor where the dimensions should be ordered as [batch_size, height, width, channels].</p><ul><li><strong>batch_size</strong>: This is the amount of data (for example, inputs such as, images, and words) in a single batch of data. We normally process data in batches as often large datasets are used for learning. At a given training step, we randomly sample a small batch of data that approximately represents the full dataset. And doing this for many steps allows us to approximate the full dataset quite well. This batch_size parameter is the same as the one we discussed in the TensorFlow input pipeline example.</li><li><strong>height and width</strong>: This is the height and the width of the input.</li><li><strong>channels</strong>: This is the depth of an input (for example, for a RGB image, channels will be 3 — a channel for each color).</li></ul><p><strong>filter</strong>: This is a 4D tensor that represents the convolution window of the convolution operation. The filter dimensions should be [height, width, in_channels, out_channels]:</p><ul><li><strong>height and width</strong>: This is the height and the width of the filter (often smaller than that of the input)</li><li><strong>in_channels</strong>: This is the number of channels of the input to the layer</li><li><strong>out_channels</strong>: This is the number of channels to be produced in the output of the layer</li></ul><p><strong>strides</strong>: This is a list with four elements, where the elements are [batch_stride, height_stride, width_stride, channels_stride]. The strides argument denotes how many elements to skip during a single shift of the convolution window on the input. If you do not completely understand what strides are, you can use the default value of 1.</p><p><strong>padding</strong>: This can be one of [‘SAME’, ‘VALID’]. It decides how to handle the convolution operation near the boundaries of the input. The VALID operation performs the convolution without padding. If we were to convolve an input of <em>n </em>length with a convolution window of size <em>h</em>, this will result in an output of size <em>(n-h+1 &lt; n)</em>. The diminishing of the output size can severely limit the depth of neural networks. SAME pads zeros to the boundary such that the output will have the same height and width as the input.</p><p>To gain a better understanding of what filter size, stride, and padding are, refer to <em>Figure 2.7</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*ULFj0FXTAh2I4p-mNINxsg.png" /><figcaption>Figure 2.7: The convolution operation</figcaption></figure><p>As we mentioned before, TensorFlow 2.x also provide us a tf.keras realization of convolution operation, tf.keras.layers.Conv2D. Different from tf.nn.conv2d, tf.keras.layers.Conv2D accepts different parameters. filters no longer mean the tensor that represents the convolution window of the convolution but a number that represents how many filters do you want to use in this layer. that is, the dimensionality of the output space.kernel_size is an integer or tuple/list of 2 integers, specifying the height and width of the 2D convolution window. Here you may ask where x_filter defined in the above code goes. Well, in tf.keras, we don’t have to input an x_filter by yourself. Since in practice, x_filter is what we want to train to get, tf.keras helps us initialize it implicitly and automatically.</p><p>The following is the implementation of the convolution operation by tf.keras. You can find that the result is different from above because the x_filter is initialized by tf.keras, not by us. But the shape is still the same because we specified filters is 1 and x_filter shape is 2 x 2 just like the above code:</p><p>x = tf.constant( [[</p><p>[[1],[2],[3],[4]],</p><p>[[4],[3],[2],[1]],</p><p>[[5],[6],[7],[8]],</p><p>[[8],[7],[6],[5]]</p><p>]],</p><p>dtype=tf.float32)</p><p>x_conv_keras = tf.keras.layers.Conv2D(filters=1, kernel_size=2, padding=x_padding)(x)</p><p>print(x_conv_keras)</p><p># Returns (out) =&gt;tf.Tensor(</p><p>[[[[ 3.890316 ]</p><p>[ 2.5445178]</p><p>[ 1.1987193]]</p><p>[[ 7.4837427]</p><p>[ 8.829541 ]</p><p>[10.17534 ]]</p><p>[[ 8.851723 ]</p><p>[ 7.5059247]</p><p>[ 6.160126 ]]]], shape=(1, 3, 3, 1), dtype=float32)</p><h3>4. The pooling operation</h3><p>A pooling operation behaves similar to the convolution operation, but the final output is different. Instead of outputting the sum of the element-wise multiplication of the filter and the image patch, we now take the maximum element of the image patch for that location (see Figure 2.8):</p><p>x = tf.constant( [[</p><p>[[1],[2],[3],[4]],</p><p>[[4],[3],[2],[1]],</p><p>[[5],[6],[7],[8]],</p><p>[[8],[7],[6],[5]]</p><p>]],</p><p>dtype=tf.float32)</p><p>x_ksize = [1,2,2,1]</p><p>x_stride = [1,2,2,1]</p><p>x_padding = ‘VALID’</p><p>x_pool = tf.nn.max_pool(input=x, ksize=x_ksize, strides=x_stride, padding=x_padding)</p><p>or</p><p>x_pool_keras = tf.keras.layers.MaxPool2D()(x)</p><p># Returns (out) =&gt;</p><p>[[[[ 4.]</p><p>[ 4.]],</p><p>[[ 8.]</p><p>[ 8.]]]]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/348/1*TbKqDxBoy8sImdhQXWyPVg.png" /><figcaption>Figure 2.8: The max-pooling operation</figcaption></figure><p><strong>Other Series of Understanding TensorFlow:</strong></p><p>Serie1: <a href="https://medium.com/@Adline125/understanding-tensorflow-series-979e71cc5562">https://medium.com/@Adline125/understanding-tensorflow-series-979e71cc5562</a></p><p>Serie2: <a href="https://medium.com/@Adline125/understanding-tensorflow-fcc431891d08">https://medium.com/@Adline125/understanding-tensorflow-fcc431891d08</a></p><p>Serie3–1: <a href="https://medium.com/@Adline125/understanding-tensorflow-ce18f0e1bbbc">https://medium.com/@Adline125/understanding-tensorflow-ce18f0e1bbbc</a></p><p>Serie3–2: <a href="https://medium.com/@Adline125/understanding-tensorflow-2c6496b71368">https://medium.com/@Adline125/understanding-tensorflow-2c6496b71368</a></p><p>Serie4: <a href="https://medium.com/@Adline125/understanding-tensorflow-94bdea8e1fd9">https://medium.com/@Adline125/understanding-tensorflow-94bdea8e1fd9</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce18f0e1bbbc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding TensorFlow: Part 2]]></title>
            <link>https://medium.com/@Adline125/understanding-tensorflow-fcc431891d08?source=rss-e25490321c4b------2</link>
            <guid isPermaLink="false">https://medium.com/p/fcc431891d08</guid>
            <dc:creator><![CDATA[dan lee]]></dc:creator>
            <pubDate>Thu, 09 Sep 2021 11:45:01 GMT</pubDate>
            <atom:updated>2021-10-31T05:24:09.005Z</atom:updated>
            <content:encoded><![CDATA[<h4>Serie 2: Basic operations in TensorFlow 2.x</h4><p>Here is today’s outline:</p><ol><li><strong>Tensor creation</strong></li></ol><ul><li><strong>Create Tensors by Python basic types and Numpy array</strong></li><li><strong>Create Tensors by tf.random</strong></li><li><strong>Create Tensors by special value function of TensorFlow</strong></li></ul><p><strong>2. Tensor merge and split</strong></p><p><strong>3. Tensor comparison and sort</strong></p><p><strong>4. Mathematical operations</strong></p><h3>1. Tensor creation</h3><p>The data type in TensorFlow is Tensor. A scalar is a tensor, a vector is a tensor, a matrix is a tensor, and the matrix of a matrix is a tensor, too. Tensor is a compound type. The elements in Tensor are python basic data types that we are quite familiar with, including bool, int, float, double, string, array.</p><p>TensorFlow 2.x is also quite friendly to Numpy. It perfectly supports Data type switch between Tensor and Numpy.</p><p>Let’s use examples to create tensors and make it more clear.</p><h3>1.1 Create Tensors by Python basic types and Numpy array</h3><p>import tensorflow as tf</p><p>import numpy as np</p><p># int</p><p>print(tf.constant(5))</p><p># float</p><p>print(tf.constant(5.0))</p><p># double</p><p>print(tf.constant(5.0, dtype=tf.float64))</p><p># bool</p><p>print(tf.constant(True))</p><p># string</p><p>print(tf.constant(‘hello, world’))</p><p># python list</p><p>print(tf.constant([1., 2., 3.]))</p><p># numpy list</p><p>print(tf.constant(np.array([1., 2., 3.])))</p><p># Returns (out) =&gt;</p><p>tf.Tensor(5, shape=(), dtype=int32)</p><p>tf.Tensor(5.0, shape=(), dtype=float32)</p><p>tf.Tensor(5.0, shape=(), dtype=float64)</p><p>tf.Tensor(True, shape=(), dtype=bool)</p><p>tf.Tensor(b’tensorflow’, shape=(), dtype=string)</p><p>tf.Tensor([1. 2. 3.], shape=(3,), dtype=float32)</p><p>tf.Tensor([1. 2. 3.], shape=(3,), dtype=float64)</p><p>Note that the dtype of Tensors created by python list and numpy list are different: float64 and float32. We should keep in mind that the default basic data type that switches from Numpy to Tensor is double. GPU processes float32 is much faster than processing double(float64). If you plan to run your code on GPU, you’d better explicitly point out the dtype should be float32 here by this:</p><p>print(tf.constant(np.array([1., 2., 3.]), dtype=tf.float32))</p><p># Returns (out) =&gt;</p><p>tf.Tensor([1. 2. 3.], shape=(3,), dtype=float32)</p><p>tf.convert_to_tensor can also convert other data types to Tensor. One example is given below, and you can try other data types by yourself. Definitely you would get the same result in both ways. tf.covert_to tensor is often used after we vectorize the x_data to turn its data type into Tensor.</p><p>print(tf.constant([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]]))</p><p>print(tf.convert_to_tensor([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]]))</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[0.1 0.2 0.3]</p><p>[0.4 0.5 0.6]], shape=(2, 3), dtype=float32)</p><p>tf.Tensor(</p><p>[[0.1 0.2 0.3]</p><p>[0.4 0.5 0.6]], shape=(2, 3), dtype=float32)</p><h3>1.2 Create Tensors by tf.random</h3><p>tf.random.normal: Outputs random values from a normal distribution. The default mean is 0.0. the default standard deviation is 1.0. The shape parameter need to be explicitly assigned.</p><p>import tensorflow as tf</p><p># tf.random.normal.</p><p>print(tf.random.normal([2, 2], mean=0, stddev=1))</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[ 0.07503338 0.690836 ]</p><p>[-0.54832345 -0.8049861 ]], shape=(2, 2), dtype=float32)</p><p>tf.random.truncated_normal: The generated values follow a normal distribution with specified mean and standard deviation, except that values whose magnitude is more than 2 standard deviations from the mean are dropped and re-picked.</p><p># tf.random.truncated_normal. Outputs random values from a truncated normal</p><p># distribution.</p><p>print(tf.random.truncated_normal([2,2], mean=0, stddev=1, dtype=tf.double))</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[ 0.77151253 -0.10534472]</p><p>[ 0.33723833 -0.29132416]], shape=(2, 2), dtype=float64)</p><p>tf.random.uniform: The generated values follow a uniform distribution in the range [minval, maxval). The lower bound minval is included in the range, while the upper bound maxval is excluded.For floats, the default range is [0, 1). For ints, at least maxval must be specified explicitly.</p><p># tf.random.uniform. Outputs random values from a uniform distribution.</p><p>print(tf.random.uniform([2, 2], minval=0, maxval=1))</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[0.37484133 0.54192734]</p><p>[0.8302536 0.4088725 ]], shape=(2, 2), dtype=float32)</p><h3>1.3 Create Tensors by special value function of TensorFlow</h3><p>tf.ones: Create a tensor with all elements set to 1. This operation returns a dtype tensor with a shape of all elements are set to 1.</p><p># tf.ones</p><p>print(tf.ones(shape=(1,3),dtype=tf.float32))</p><p># Returns (out) =&gt;</p><p>tf.Tensor([[1. 1. 1.]], shape=(1, 3), dtype=float32)</p><p>tf.zeros: Create a tensor with all elements set to 0. This operation returns a dtype tensor with a shape of all elements are set to 0.</p><p># tf.zeors</p><p>print(tf.zeros(shape=(1,3),dtype=tf.float32))</p><p># Returns (out) =&gt;</p><p>tf.Tensor([[0. 0. 0.]], shape=(1, 3), dtype=float32)</p><p>tf.eye: Construct an identity matrix or a batch of matrices.</p><p># tf.eye</p><p>print(tf.eye(4, dtype=tf.float32))</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[1. 0. 0. 0.]</p><p>[0. 1. 0. 0.]</p><p>[0. 0. 1. 0.]</p><p>[0. 0. 0. 1.]], shape=(4, 4), dtype=float32)</p><p>tf.ones_like: Create a tensor with all elements set to 1. Given a tensor, this operation returns a tensor of the same type and shape as the tensor with all elements set to 1. Optionally, it can specify the tensor with a new type (dtype).</p><p># tf.ones_like</p><p>print(tf.ones_like(tf.linspace(0.0, 1.0, 4)))</p><p># Returns (out) =&gt;</p><p>tf.Tensor([1. 1. 1. 1.], shape=(4,), dtype=float32)</p><p>tf. zeros_like: Create a tensor with all elements set to 0. Given a tensor, this operation returns a tensor of the same type and shape as the tensor with all elements set to 0. Optionally, it can specify the tensor with a new type (dtype).</p><p># tf.zeros_like</p><p>print(tf.zeros_like(tf.linspace(0.0, 1.0, 4)))</p><p># Returns (out) =&gt;</p><p>tf.Tensor([0. 0. 0. 0.], shape=(4,), dtype=float32)</p><p>tf. linspace: An calculation instruction that evenly divides the given ‘start’ and ‘stop’ numbers by ‘num’ and generates values in the interval.</p><p># tf.linspace</p><p>print(tf.linspace(start=0.0,stop=1.0, num=3))</p><p># Returns (out) =&gt;</p><p>tf.Tensor([0. 0.5 1. ], shape=(3,), dtype=float32)</p><p>tf.fill: Creates a tensor filled with a scalar value. This operation creates a tensor of shape dims and fills it with value.</p><p># tf.fill</p><p>print(tf.fill(dims=(1,5), value=1.0))</p><p># Returns (out) =&gt;</p><p>tf.Tensor([[1. 1. 1. 1. 1.]], shape=(1, 5), dtype=float32)</p><h3>2 Tensor merge and split</h3><p>Merging refers to merging multiple tensors into one tensor in a certain dimension. There are two kinds of Merging-concatenation and stacking. Concatenation does not create new dimensions, but stacking creates new dimensions.</p><p>The concatenation can be performed on any dimension. The only constraint is that the length of the non-concatenated dimensions must be the same.</p><p>a = tf.random.normal([1,2,3])</p><p>b = tf.random.normal([2,2,3])</p><p>c = tf.concat([a,b], axis=0)</p><p>print(c)</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[[-0.27534676 -0.16034059 0.52429354]</p><p>[-0.5917587 -0.23074977 -1.3515841 ]]</p><p>[[-0.4800986 -1.7884752 -0.75610226]</p><p>[-1.7305112 -0.7396962 1.0428202 ]]</p><p>[[-0.33405465 0.3331637 -0.26588857]</p><p>[-0.19221431 -1.5180566 -0.67606276]]], shape=(3, 2, 3), dtype=float32)</p><p>tf.concat directly merges data on existing dimensions and does not create a new dimension. If you want to create a new dimension when merging data, you need to use the tf.stack. The parameter axis specifies where to insert the new dimension. When an axis is a positive number, it will insert a new dimension in front of axis. When the axis is negative, it will insert a new dimension in the next position.</p><p>tf.stack also needs to meet the merging conditions of tensor stacking. It requires all merged tensors to have the same shape before they are merged.</p><p>a = tf.random.normal([2,3])</p><p>b = tf.random.normal([2,3])</p><p>c = tf.stack([a,b], axis = 0)</p><p>print(c)</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[[ 0.7345577 0.6821109 1.2609174 ]</p><p>[ 1.6845615 0.7451898 0.04291777]]</p><p>[[-0.33740672 0.71333706 1.2531255 ]</p><p>[ 0.66583854 -0.8219059 0.278419 ]]], shape=(2, 2, 3), dtype=float32)</p><p>d = tf.stack([a,b], axis = -1)</p><p>print(d)</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[[ 0.53225416 1.041896 ]</p><p>[-0.66618955 0.03342717]</p><p>[-1.6268841 0.18227476]]</p><p>[[ 0.11992308 0.33611476]</p><p>[ 0.14574063 -0.13129552]</p><p>[-0.59104127 1.382147 ]]], shape=(2, 3, 2), dtype=float32)</p><p>The inverse process of the merge operation is splitting. splitting is to split a tensor into multiple tensors. There are two kinds of split-split and unstack. Pay attention to the output shape.</p><p>#split</p><p>x = tf.random.normal([3,2,3])</p><p>print(x)</p><p>result = tf.split(x, axis=0, num_or_size_splits =3)</p><p>print(result)</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[[ 0.3446128 1.8321378 -0.3682034 ]</p><p>[-0.59962803 -1.3437859 -0.20841458]]</p><p>[[-0.28618914 -0.37867725 -2.6242352 ]</p><p>[-1.4301056 -2.365182 -0.52069366]]</p><p>[[-0.5681175 2.4599526 -0.03526611]</p><p>[ 0.7553041 1.8161643 -0.7825681 ]]], shape=(3, 2, 3), dtype=float32)</p><p>[&lt;tf.Tensor: shape=(1, 2, 3), dtype=float32, numpy=</p><p>array([[[ 0.3446128 , 1.8321378 , -0.3682034 ],</p><p>[-0.59962803, -1.3437859 , -0.20841458]]], dtype=float32)&gt;,</p><p>&lt;tf.Tensor: shape=(1, 2, 3), dtype=float32, numpy=</p><p>array([[[-0.28618914, -0.37867725, -2.6242352 ],</p><p>[-1.4301056 , -2.365182 , -0.52069366]]], dtype=float32)&gt;,</p><p>&lt;tf.Tensor: shape=(1, 2, 3), dtype=float32, numpy=</p><p>array([[[-0.5681175 , 2.4599526 , -0.03526611],</p><p>[ 0.7553041 , 1.8161643 , -0.7825681 ]]], dtype=float32)&gt;]</p><p>#unstack</p><p>x = tf.random.normal([3,2,3])</p><p>print(x)</p><p>result = tf.unstack(x,axis = 0)</p><p>print(result)</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[[-0.45672733 -0.6107957 -1.5969335 ]</p><p>[-0.05063461 0.90097845 -0.51631683]]</p><p>[[ 1.440197 -0.28877714 -0.52491015]</p><p>[-2.1945505 0.01744763 0.35254684]]</p><p>[[-0.66387564 0.63262576 -0.14352717]</p><p>[-0.08475702 -1.186847 0.7461847 ]]], shape=(3, 2, 3), dtype=float32)</p><p>[&lt;tf.Tensor: shape=(2, 3), dtype=float32, numpy=</p><p>array([[-0.45672733, -0.6107957 , -1.5969335 ],</p><p>[-0.05063461, 0.90097845, -0.51631683]], dtype=float32)&gt;,</p><p>&lt;tf.Tensor: shape=(2, 3), dtype=float32, numpy=</p><p>array([[ 1.440197 , -0.28877714, -0.52491015],</p><p>[-2.1945505 , 0.01744763, 0.35254684]], dtype=float32)&gt;,</p><p>&lt;tf.Tensor: shape=(2, 3), dtype=float32, numpy=</p><p>array([[-0.66387564, 0.63262576, -0.14352717],</p><p>[-0.08475702, -1.186847 , 0.7461847 ]], dtype=float32)&gt;]</p><h3>3. Tensor comparison and sort</h3><p>In actual work scenarios, we may only be interested in data that meets certain conditions, or needs to count the accuracy of a task and other indicators. Generally, we need to compare the predicted result with the true label. We need to calculate accuracy rate though the results that are the same as the labels or greater than a certain threshold. TensorFlow provides tensor comparison operations, including equal or greater or not.</p><p>y = tf.random.uniform([10],dtype=tf.int64,maxval=10)</p><p>print(y)</p><p>y_hat = tf.random.uniform([10],dtype=tf.int64,maxval=10)</p><p>print(y_hat)</p><p>result = tf.equal(y_hat,y)</p><p>print(result)</p><p>print(y_hat &gt; 5)</p><p># Returns (out) =&gt;</p><p>tf.Tensor([4 8 8 3 5 8 2 1 6 7], shape=(10,), dtype=int64)</p><p>tf.Tensor([0 2 2 2 3 9 8 2 4 8], shape=(10,), dtype=int64)</p><p>tf.Tensor([False False False False False False False False False False], shape=(10,), dtype=bool)</p><p>tf.Tensor([False False False False False True True False False True], shape=(10,), dtype=bool)</p><p>In some cases, we need data that is ranked higher. TensorFlow 2.x provides a variety of sorting functions. In addition to the ordinary ascending and descending order, there are more convenient operations, like top data and the index of the largest data.</p><p>tensor = tf.random.uniform([10], maxval=10)</p><p>print(tensor)</p><p># sort</p><p>print(tf.sort(tensor, direction=’DESCENDING’))</p><p># indice of sort result</p><p>print(tf.argsort(tensor, direction=’DESCENDING’))</p><p># top k</p><p>print(tf.math.top_k(a, 3))</p><p># set all elements greater than 4</p><p>print(tf.maximum(a, 5))</p><p># set all elements smaller than 4</p><p>print(tf.minimum(a, 5))</p><p># set all elements greater than 3 smaller than 7</p><p>print(tf.clip_by_value(a, 3, 7))</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[8.151581 2.8347158 8.287186 1.431247 5.478798 4.6553183 5.820148</p><p>4.525962 7.1494665 6.572027 ], shape=(10,), dtype=float32)</p><p>tf.Tensor(</p><p>[8.287186 8.151581 7.1494665 6.572027 5.820148 5.478798 4.6553183</p><p>4.525962 2.8347158 1.431247 ], shape=(10,), dtype=float32)</p><p>tf.Tensor([2 0 8 9 6 4 5 7 1 3], shape=(10,), dtype=int32)</p><p>TopKV2(values=&lt;tf.Tensor: shape=(3,), dtype=float32, numpy=array([8.287186 , 8.151581 , 7.1494665], dtype=float32)&gt;, indices=&lt;tf.Tensor: shape=(3,), dtype=int32, numpy=array([2, 0, 8], dtype=int32)&gt;)</p><p>tf.Tensor(</p><p>[8.151581 5. 8.287186 5. 5.478798 5. 5.820148</p><p>5. 7.1494665 6.572027 ], shape=(10,), dtype=float32)</p><p>tf.Tensor(</p><p>[5. 2.8347158 5. 1.431247 5. 4.6553183 5.</p><p>4.525962 5. 5. ], shape=(10,), dtype=float32)</p><p>tf.Tensor(</p><p>[7. 3. 7. 3. 5.478798 4.6553183 5.820148</p><p>4.525962 7. 6.572027 ], shape=(10,), dtype=float32)</p><p>In big data processing, what we often care about is not each element’s value, but the average, maximum, minimum, and corresponding index in a certain dimension (default global dimension).</p><p>tensor = tf.constant([[1,2],[3,4]], dtype=tf.float32)</p><p># max in dimension 1</p><p>print(tf.reduce_max(tensor, axis=1))</p><p># min in dimension 0</p><p>print(tf.reduce_min(tensor, axis=0))</p><p># average, default global dimension</p><p>print(tf.reduce_mean(tensor))</p><p># index of max value in dimension 0</p><p>print(tf.argmax(tensor, axis=0))</p><p># index of min value in dimension 1</p><p>print(tf.argmin(tensor, axis=1))</p><p># Returns (out) =&gt;</p><p>tf.Tensor([2. 4.], shape=(2,), dtype=float32)</p><p>tf.Tensor([1. 2.], shape=(2,), dtype=float32)</p><p>tf.Tensor(2.5, shape=(), dtype=float32)</p><p>tf.Tensor([1 1], shape=(2,), dtype=int64)</p><p>tf.Tensor([0 0], shape=(2,), dtype=int64)</p><p>One more example, after we use the loss function getting each sample’s loss, the average loss of the samples needs to be calculated.</p><p>y_hat = tf.random.normal([10,5])</p><p>y = tf.constant([1, 2, 3, 4, 5, 1, 2, 3, 4, 5])</p><p>y = tf.one_hot(y,depth=5)</p><p>loss = tf.keras.losses.mse(y,y_hat)</p><p>loss_mean = tf.reduce_mean(loss)</p><p>print(loss_mean)</p><p># Returns (out) =&gt;</p><p>tf.Tensor(1.4730119, shape=(), dtype=float32)</p><p>Of course, you can also get the maximum and minimum loss, even their indices which indicates the sample with maximum loss or minimum loss.</p><p>loss_max = tf.reduce_max(loss)</p><p>loss_min = tf.reduce_min(loss)</p><p>idx_max = tf.argmax(loss)</p><p>idx_min = tf.argmin(loss)</p><p>print(loss)</p><p>print(loss_max)</p><p>print(loss_min)</p><p>print(idx_max)</p><p>print(idx_min)</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[0.665666 0.50770426 0.37840793 0.550116 0.674733 0.5545435</p><p>2.1785192 2.1507242 0.500903 1.2179064 ], shape=(10,), dtype=float32)</p><p>tf.Tensor(2.1785192, shape=(), dtype=float32)</p><p>tf.Tensor(0.37840793, shape=(), dtype=float32)</p><p>tf.Tensor(6, shape=(), dtype=int64)</p><p>tf.Tensor(2, shape=(), dtype=int64)</p><h3>4. Mathematical operations</h3><p>TensorFlow 2.x support mathematical operations much more friendly that TensorFlow 1.x. The commonly used operations are listed below. It is clear and absolutely meets python programming habit.</p><p>a = tf.constant([[1,2],[3,4]], dtype=tf.float32)</p><p>b = tf.constant([[5,6],[7,8]], dtype=tf.float32)</p><p># element-wise add</p><p>print(a+b)</p><p># Returns (out) =&gt;</p><p>f.Tensor(</p><p>[[ 6. 8.]</p><p>[10. 12.]], shape=(2, 2), dtype=float32)</p><p># element-wise minus</p><p>print(a-b)</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[-4. -4.]</p><p>[-4. -4.]], shape=(2, 2), dtype=float32)</p><p># element-wise multipy</p><p>print(a*b)</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[ 5. 12.]</p><p>[21. 32.]], shape=(2, 2), dtype=float32)</p><p># element-wise divide</p><p>print(‘a/b=’, a/b)</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[0.2 0.33333334]</p><p>[0.42857143 0.5 ]], shape=(2, 2), dtype=float32)</p><p># element-wise floor division</p><p>print(‘a//b’, a//b)</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[0. 0.]</p><p>[0. 0.]], shape=(2, 2), dtype=float32)</p><p># element-wise remainder</p><p>print(‘b%a=’, b%a)</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[0. 0.]</p><p>[1. 0.]], shape=(2, 2), dtype=float32)</p><p># square</p><p>print(a**2)</p><p># square</p><p>print(tf.pow(a, 2.0))</p><p># square</p><p>print(tf.square(a))</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[ 1. 4.]</p><p>[ 9. 16.]], shape=(2, 2), dtype=float32)</p><p>tf.Tensor(</p><p>[[ 1. 4.]</p><p>[ 9. 16.]], shape=(2, 2), dtype=float32)</p><p>tf.Tensor(</p><p>[[ 1. 4.]</p><p>[ 9. 16.]], shape=(2, 2), dtype=float32)</p><p># square root</p><p>print(tf.sqrt(a))</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[1. 1.4142135]</p><p>[1.7320508 2. ]], shape=(2, 2), dtype=float32)</p><p># exp</p><p>print(tf.exp(a))</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[ 2.7182817 7.389056 ]</p><p>[20.085537 54.59815 ]], shape=(2, 2), dtype=float32)</p><p># log</p><p>print(tf.math.log(a))</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[0. 0.6931472]</p><p>[1.0986123 1.3862944]], shape=(2, 2), dtype=float32)</p><p># maxtrix multipy</p><p>print(tf.matmul(a, b))</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[19. 22.]</p><p>[43. 50.]], shape=(2, 2), dtype=float32)</p><p><strong>Other series of Understanding TensorFlow:</strong></p><p>Serie1: <a href="https://medium.com/@Adline125/understanding-tensorflow-series-979e71cc5562">https://medium.com/@Adline125/understanding-tensorflow-series-979e71cc5562</a></p><p>Serie2: <a href="https://medium.com/@Adline125/understanding-tensorflow-fcc431891d08">https://medium.com/@Adline125/understanding-tensorflow-fcc431891d08</a></p><p>Serie3–1: <a href="https://medium.com/@Adline125/understanding-tensorflow-ce18f0e1bbbc">https://medium.com/@Adline125/understanding-tensorflow-ce18f0e1bbbc</a></p><p>Serie3–2: <a href="https://medium.com/@Adline125/understanding-tensorflow-2c6496b71368">https://medium.com/@Adline125/understanding-tensorflow-2c6496b71368</a></p><p>Serie4: <a href="https://medium.com/@Adline125/understanding-tensorflow-94bdea8e1fd9">https://medium.com/@Adline125/understanding-tensorflow-94bdea8e1fd9</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fcc431891d08" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding TensorFlow Series]]></title>
            <link>https://medium.com/@Adline125/understanding-tensorflow-series-979e71cc5562?source=rss-e25490321c4b------2</link>
            <guid isPermaLink="false">https://medium.com/p/979e71cc5562</guid>
            <dc:creator><![CDATA[dan lee]]></dc:creator>
            <pubDate>Thu, 09 Sep 2021 11:31:25 GMT</pubDate>
            <atom:updated>2021-10-31T05:23:40.127Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding TensorFlow: Part 1</h3><h4>Serie 1: TensorFlow 1.x VS TensorFlow 2.x</h4><p>In this article series, you will get an in-depth understanding of TensorFlow. This is an open-source distributed numerical computation framework.</p><p>We will get started today with TensorFlow by defining a simple calculation and trying to compute it using both Tensorflow 1.x and TensorFlow 2.x aiming to have an idea of how convenient Tensorflow 2.x is now comparing with Tensorflow 1.x.</p><p>After we successfully complete this, we will investigate the eager mode and the autograph mode of TensorFlow 2.x.</p><p>Here is the content outline:</p><ul><li>TensorFlow 1.x VS TensorFlow 2.x</li><li>Eager mode of TensorFlow 2.x</li><li>Autograph mode of TensorFlow 2.x</li></ul><h4>TensorFlow 1.x VS TensorFlow 2.x</h4><p>TensorFlow has developed two phases by now-TensorFlow 1.x and TensorFlow 2.x. Our introduction will focus on Tensorflow 2.x. Before that, it is better for us to have an idea about these two versions’ main differences so that we can have a focus to learn the features of TensorFlow 2.x and how convenient TensorFlow 2.x becomes, comparing with Tensorflow 1.x.</p><p>Let’s take a look at the comparison with a simple example first. We are now using TensorFlow 1.x and Tensorflow 2.x to perform the ‘add’ computation individually.</p><p>c = a + b</p><p>TensorFlow 1.x’s code:</p><p>import tensorflow as tf</p><p>#build graph</p><p>a = tf.constant(3)</p><p>b = tf.constant(4)</p><p>c = a + b</p><p>#use session to run</p><p>with tf.Session() as sess:</p><p>result = sess.run(c)</p><p>print(result)</p><p>sess.close()</p><p>In TensorFlow 1.x, there are at least two main parts to build in your code even with the simplest ‘add’ computation. The first step is to create a static computation graph. In the code above, the ‘#build graph’ block is to fulfill this job. It doesn’t actually perform any computation, even though it looks like it does. To actually do the computation, you need to open a TensorFlow session. Here comes the second step-the ‘#use session to run’ block. The tf.Session() will place the operations onto devices like CPUs or GPUs and running them. If there are variables in the computation graph, the tf.Session() would hold those variables, initialize, and evaluates them, too.</p><p>We can see that the whole process is not instinctive and violate our programming habit. especially when you want to debug a larger neural network, it would be a disaster. TensorFlow 2.x comes to save us from that.</p><p>TensorFlow 2.x’s code:</p><p>import tensorflow as tf</p><p>a = tf.constant(3.)</p><p>b = tf.constant(5.)</p><p>c = a + c</p><p>print(c)</p><p>Yes, it is so easy! To finish the ‘add’ computation, TensorFlow 2.x only needs three lines. The first and second line is to create elements in ‘add’ computation and the third line is to execute the ‘add’ operation and return the result directly. Different from TensorFlow 1.x using the static graph, TensorFlow 2.x is dynamic. We can check the computation result directly after each line of your code. It would be much easier to debug. And it adapted to our programming habit.</p><p>Compared with TensorFlow 1.x, TensorFlow 2.x makes the development of ML applications much easier. With tight integration of Keras into TensorFlow, eager execution by default, and Pythonic function execution, TensorFlow 2.x makes the experience of developing applications as familiar as possible for Python developers.</p><h3>Eager mode of TensorFlow 2.x</h3><p>TensorFlow 2.x removes the unintuitive way to build a computational graph and run it later in the tf.session. The key to realizing it is eager execution. Eager execution provides us an imperative programming environment with Python data structures, immediate error reporting, and Python flow control.</p><p>All three of them changed TensorFlow operation behavior to evaluate and return values to Python immediately. It has concrete value flow naturally in your code structure and makes standard Python debugging possible.</p><p>Eager execution supports most TensorFlow operations and GPU acceleration. In Tensorflow 2.x, eager execution is enabled by default.</p><p>Let’s use a simple example to run TensorFlow operations and of course, you will see the result will return immediately.</p><p>h = sigmoid(W * x + b)</p><p>Please note that the examples in this section is only for giving you an impression that how tensorflow 2.x code looks like. The details of each kind of Tensor creation and operation will be introduced step by step in next section.</p><p>import tensorflow as tf</p><p>x = tf.constant([[1., 2., 3.], [4., 5., 6.]])</p><p>w = tf.Variable(tf.random.normal([3, 2]))</p><p>b = tf.Variable(tf.random.normal([2]))</p><p>z = tf.matmul(x, w) + b</p><p>y = tf.nn.sigmoid(z)</p><p>print(y)</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[0.4677037 0.25148553]</p><p>[0.9712641 0.0902136 ]], shape=(2, 2), dtype=float32)</p><p>The static computational graph and session all disappear in our code! We can easily check out the current result at runtime simply by print(). Of course, you can put print() right after the operation line whenever you want to see the intermediate result. For example, put print(x) right after x = tf.constant([[1., 2., 3.], [4., 5., 6.]]) to check x value.</p><p>To have a more general impression of how to use eager execution to work, let’s use eager execution to finish a whole process of a Linear model training. The purpose is not to grasp every detail right now right here but to get you familiar with the TensorFlow 2.x style programming. It doesn’t matter if you can’t fully understand what is going on in the code. Either briefly read it through to get a basic idea or skip this part then come back after reading more about TensorFlow and models, both are good.</p><p>First, let’s fake some data for training. In this example, we generate 2000 samples. Each sample is a vector with 10 elements in it. The true parameters that we want to finally have are w_true and b_true. Each element in w_true is set to be 3 with the shape [10, 5]. Each element in b_true is set to be 1 with the shape [5].</p><p>NUM_EXAMPLES = 2000</p><p>x = tf.random.normal([NUM_EXAMPLES, 10])</p><p>noise = tf.random.normal([NUM_EXAMPLES, 5])</p><p>w_true = tf.ones([10, 5])</p><p>w_true = w_true * 3</p><p>b_true = tf.ones([5])</p><p>y = tf.matmul(x, w_true) + b_true + noise</p><p>print(f’x sample: {x[0:1]}’)</p><p>print(f’y sample: {y[0:1]}’)</p><p># Returns (out) =&gt;</p><p>x sample: [[ 0.7714453 0.96156746 1.2024771 -1.0653574 -0.749367 2.1572323</p><p>-1.1070591 -0.13671888 0.7504565 1.1920955 ]]</p><p>y sample: [[14.045872 12.701928 13.449406 12.854302 12.903345]]</p><p>Next, build the linear model with the tf.keras model subclass. Model parameters are W and B. The linear model returns w*x + b.</p><p>class Linear(tf.keras.Model):</p><p>def __init__(self):</p><p>super(Linear, self).__init__()</p><p>self.W = tf.Variable(tf.random.normal([10, 5]), name=’weight’)</p><p>self.B = tf.Variable(tf.random.normal([5]), name=’bias’)</p><p>def call(self, inputs):</p><p>return tf.matmul(inputs, self.W) + self.B</p><p>Third, create the model and the optimizer.</p><p>model = Linear()</p><p>optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)</p><p>Fourth, train the model. After training the model, we can fix our model parameters self.W and self.B. Their values are listed in the output. It is a demo code, the W and B are not close enough to the w_true and b_true. But we can still see the trends that they are bumping around with the target as their center.</p><p>steps = 100</p><p>for i in range(steps):</p><p>with tf.GradientTape() as tape:</p><p>#loss</p><p>error = model(x) — y</p><p>loss_value = tf.reduce_mean(tf.square(error))</p><p>grads = tape.gradient(loss_value, [model.W, model.B])</p><p>optimizer.apply_gradients(zip(grads, [model.W, model.B]))</p><p>if i % 20 == 0:</p><p>print(f”Loss at step {i}: {loss_value}”)</p><p>print(f”W = {model.W.numpy()}, B = {model.B.numpy()}”)</p><p># Returns (out) =&gt;</p><p>Loss at step 0: 0.9959805607795715</p><p>Loss at step 20: 0.9947214722633362</p><p>Loss at step 40: 0.993640124797821</p><p>Loss at step 60: 0.9927115440368652</p><p>Loss at step 80: 0.9919135570526123</p><p>W = [[3.013761 2.972824 2.9576807 2.9959006 2.9828606] [2.9921997 2.9522212 2.9901388 2.981263 2.9870505] [2.983197 2.9589314 3.0155313 3.0105908 3.0207992] [3.002246 2.9900434 2.9911227 3.0125127 2.9714382] [2.972854 2.9398937 3.0158775 2.9727569 3.0097048] [3.0214002 2.9796052 3.002326 3.0160155 3.0155087] [2.9566758 2.9727437 2.9714837 2.9215772 2.9489958] [2.9522686 3.001058 2.9785402 2.97918 3.0170207] [2.9413743 2.9621522 2.9582999 2.9600418 2.9680848] [2.970826 2.986206 2.9471984 2.9669378 2.975441 ]], B = [1.0471543 0.98861533 1.0299664 0.9623249 0.96185046]</p><p>The final step, save, load and predict.</p><p># save</p><p>model.save_weights(‘weights’)</p><p>del model</p><p># load</p><p>model = Linear()</p><p>model.load_weights(‘weights’)</p><p># predict</p><p>x_new = tf.random.normal([1, 10])</p><p>print(x_new)</p><p>print(model.predict(x_new))</p><p># Returns (out) =&gt;</p><p>tf.Tensor(</p><p>[[-0.45371628 -1.0948718 -0.5971005 -1.6125325 0.8018683 -0.4925114</p><p>-2.4727316 -0.41909572 0.96381396 -0.81628186]], shape=(1, 10), dtype=float32)</p><p>[[-13.022452 -11.254985 -9.975083 -14.200058 -10.875286]]</p><h3>Autograph mode of TensorFlow 2.x</h3><p>While eager execution makes development and debugging more interactive, TensorFlow 1.x style graph execution has advantages for distributed training, performance optimizations, and production deployment. To bridge this gap, TensorFlow 2.x introduces functions via the <a href="https://www.tensorflow.org/api_docs/python/tf/function">tf.function</a> API.</p><p>TensorFlow 2.x supports automatically construct graphs when you need them. The way you create a graph in TensorFlow 2.x is to use <a href="https://www.tensorflow.org/api_docs/python/tf/function">tf.function</a>, either as a direct call or as a decorator.</p><p>It is common to use tf.function decorator to realize autograph mode of TensorFlow 2.x. tf.function allows to write graphics code using natural Python syntax. When a function is annotated with tf.function, it can be called like any other function. It will be compiled into a graph, which means faster execution and better running on GPU or TPU.</p><p>@tf.function</p><p>def sigmoid_layer(x, w):</p><p>return tf.nn.sigmoid(tf.matmul(x, w))</p><p>x = tf.random.uniform((2, 5))</p><p>w = tf.random.uniform((5, 2))</p><p>sigmoid_layer(x, w)</p><p># Returns (out) =&gt;</p><p>&lt;tf.Tensor: shape=(2, 2), dtype=float32, numpy=</p><p>array([[0.79586333, 0.8243984 ],</p><p>[0.83552325, 0.78892213]], dtype=float32)&gt;</p><p>If the code uses multiple functions, you don’t need to annotate them all-any function called from an annotated function will also run in graphical mode. For example,</p><p>def inner_function(x, y, b):</p><p>x = tf.matmul(x, y)</p><p>x = x + b</p><p>return x</p><p>@tf.function</p><p>def outer_function(x):</p><p>y = tf.constant([[2.0], [3.0]])</p><p>b = tf.constant(4.0)</p><p>return inner_function(x, y, b)</p><p>outer_function(tf.constant([[1.0, 2.0]])).numpy()</p><p># Returns (out) =&gt;</p><p>array([[12.]], dtype=float32)</p><p>You can also decorate a custom tf.keras model by annotating the call function of the model.</p><p>class CustomModel(tf.keras.models.Model):</p><p>@tf.function</p><p>def call(self, input_data):</p><p>if tf.reduce_mean(input_data) &gt; 0:</p><p>return input_data</p><p>else:</p><p>return input_data // 2</p><p>model = CustomModel()</p><p>model(tf.constant([-2, -4]))</p><p># Returns (out) =&gt;</p><p>&lt;tf.Tensor: shape=(2,), dtype=int32, numpy=array([-1, -2], dtype=int32)&gt;</p><p><strong>Other series of Understanding TensorFlow:</strong></p><p>Serie1: <a href="https://medium.com/@Adline125/understanding-tensorflow-series-979e71cc5562">https://medium.com/@Adline125/understanding-tensorflow-series-979e71cc5562</a></p><p>Serie2: <a href="https://medium.com/@Adline125/understanding-tensorflow-fcc431891d08">https://medium.com/@Adline125/understanding-tensorflow-fcc431891d08</a></p><p>Serie3–1: <a href="https://medium.com/@Adline125/understanding-tensorflow-ce18f0e1bbbc">https://medium.com/@Adline125/understanding-tensorflow-ce18f0e1bbbc</a></p><p>Serie3–2: <a href="https://medium.com/@Adline125/understanding-tensorflow-2c6496b71368">https://medium.com/@Adline125/understanding-tensorflow-2c6496b71368</a></p><p>Serie4: <a href="https://medium.com/@Adline125/understanding-tensorflow-94bdea8e1fd9">https://medium.com/@Adline125/understanding-tensorflow-94bdea8e1fd9</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=979e71cc5562" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Attention in Recommendation Systems]]></title>
            <link>https://medium.com/ai%C2%B3-theory-practice-business/attention-in-recommendation-systems-794c796a3063?source=rss-e25490321c4b------2</link>
            <guid isPermaLink="false">https://medium.com/p/794c796a3063</guid>
            <category><![CDATA[recommendations]]></category>
            <category><![CDATA[attention]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[gaming]]></category>
            <dc:creator><![CDATA[dan lee]]></dc:creator>
            <pubDate>Tue, 12 May 2020 09:58:27 GMT</pubDate>
            <atom:updated>2020-05-12T18:26:36.621Z</atom:updated>
            <content:encoded><![CDATA[<h4>Pre-Training, Part 3: Self-Attention</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EctVTBob9ZLJN2nC.png" /><figcaption><a href="http://jalammar.github.io/illustrated-transformer/">http://jalammar.github.io/illustrated-transformer/</a></figcaption></figure><p>In my last article on <a href="https://medium.com/ai%C2%B3-theory-practice-business/attention-one-key-to-making-pre-training-successful-37b1c0b7f55a">attention in pre-training</a>, I mentioned that I used the attention mechanism in an ongoing project at game services company Yodo1.</p><p>However, the form of attention I explained in <a href="https://medium.com/ai%C2%B3-theory-practice-business/attention-one-key-to-making-pre-training-successful-37b1c0b7f55a">Pre-Training Part 2</a> is more basic than the one I ended up using for this project.</p><p>The form I used is called <em>self-attention.</em></p><p>Today, I will explain why I used this form of attention and give examples from my project to show you how self-attention helps recommendation systems express the relationship between game features — such as items for purchase and attainable levels — in a more reasonable and effective way.</p><p>First: What are recommendation systems, and how do we use them in games?</p><h3>What Are Recommendation Systems?</h3><p>Recommendation systems use machine learning algorithms to predict whether users will like certain items and unless you’ve been living in the stone age, you’ve definitely made use of them.</p><p>Perhaps you’ve experienced the following: You open Google Chrome, but you’re not sure what you’re looking for. In the welcome tab, you see “articles for you” and a topic of interest catches your eye. You open the link and read the story.</p><p>How did Google know you would want to receive this bit of news? Thanks to its recommendation system.</p><h3>Recommendation Systems in Mobile Games</h3><p>Turning to the gaming industry, recommendation systems come in handy when players want to, for example, buy new equipment but don’t know exactly what they want yet.</p><p>And recommendations have a definite impact: according to our data, more than 40% of people will directly buy the equipment that has been recommended to them.</p><h3>How do recommendation systems make games better?</h3><p>By giving new players suggestions on where to start, while saving established players time by making the items they’re interested in easier to find, recommendation systems make the gaming experience easier to enjoy.</p><p>At Yodo1, our goal is to design the recommendation system in such a way that as many players as possible can benefit from it and thereby enjoy an enhanced gaming experience.</p><p>For the game I’ve been working on, we tried to create a mixed recommendation system: one that uses multiple methods and models, from multiple perspectives, to make its recommendations.</p><p>Since I’m not at liberty to disclose this game’s title, let’s refer to it as “Robot World”.</p><p>In Robot World, players build a team of robots and equip them with tools and special skills to attack other players’ headquarters. Players need to know the robots pretty well to build a unified team.</p><p>We on the developer team can see the types of robots and skills players generally prefer to use and we understand how many choices they face when trying to optimize their team.</p><p>With so many decisions to make, players need some form of guidance or complementary opinions to improve their team’s performance and increase their chances of winning.</p><h3>Problem Solving: Making a Computer Understand Complex Relationships</h3><p>To effectively guide our players in their decision making, we need to understand and quantify the delicate relationship between different characters’ skills and the equipment available to them.</p><p>To win the game, a player needs to find new characters and skills that best fit the team they’ve built.</p><p>Thus, the problem we developers must solve is how to express the relationship between various game resources in a way a computer can understand — so that we can use it in our recommendation system.</p><p>Of course, we can use records of characters and bundles that players have obtained in the past to speculate on this relationship. But which machine-learning model is most suited to the job?</p><h3>Why We Need Self-Attention</h3><p>To an extent, game character acquisition and bundle purchases have sequence features and LSTM can reflect character acquisition sequence and bundle acquisition sequence in terms of time.</p><p>We could use LSTM’s description of sequence to reflect the relationship between various game resources, but it only gives us the perspective of time and is not straightforward enough.</p><p>Then there’s CNN, which extracts local features and seems to reflect the relationship between neighboring features, but it is not flexible enough.</p><p>Combined features are an element of the recommendation system we cannot ignore. This refers to the mix of game characters, skills, and/or equipment and reflects the relationship between various game resources.</p><p>If we can rightly reflect the combination of features and relationships between them in our recommendation system, there will be an obvious improvement in the system’s performance.</p><p>What we need to achieve this is a solution that is direct and flexible.</p><p>And that’s exactly where self-attention joins the party, because it meets my requirements very well. Plus, it’s not limited by the disappearance of gradients or adjacent elements and its characteristics allow for serialization, which can increase training speed.</p><p>So that settles it: we’re going to solve the problem with self-attention.</p><h3>A Step-by-Step Guide to Using Self-Attention</h3><p>Here is how the main process of self-attention works:</p><p>(Bear in mind that in our example, features refer to game characters, skills, and equipment bundles that our players can buy.)</p><p><strong>Step 1: Get feature</strong> <strong>embedding. </strong>Obtain the embedding vector of each feature in the embedding layer.</p><p><strong>Step 2:</strong> <strong>Initialize Q, K, V matrix.</strong> Q(query) matrix, K(key) matrix, V(value) matrix are the trainable weights in the neural network. We use them to transform each feature embedding into three kinds of vectors to calculate attention weights. We can initialize the three matrices randomly and it will give us the optimized result after training iterations.</p><p><strong>Step 3: Get q, k, v vectors. </strong>Let’s take one feature, say the character “Robot1”, as an example. We get its feature embedding “Robot1 embedding vector” in step 1. Now, by multiplying the “Robot1 embedding vector” with the Q, K, V matrix in step 2, we transform the “Robot1 embedding vector” into three kinds of vectors: the “Robot1 query vector” (q1), the “Robot1 key vector” (k1), and the “Robot1 value vector” (v1). Apply this step to all the other features so you get q2, k2, v2, q3, k3, v3, … qn, kn, vn.</p><p><strong>Step 4: Calculate attention weights.</strong> For “Robot1”, its attention weights are obtained by calculating the dot product (w1) of q1, k1, the dot product (w2) of q1, k2, the dot product (w3) of q1, k3, and so on until qn, kn. Now we have weights of “Robot1” ( w1, w2, w3, … wn). These weights can reflect the relationship between “Robot1” and other features in the game.</p><p><strong>Step 5: Normalize attention weights. </strong>Dot production w1, w2, w3, … wn are not the numbers limited within 0–1. So we can use softmax to make them stay within 0–1.</p><p><strong>Step 6: Multiply attention weights with v. </strong>Do you still remember v1, v2, v3, … vn in step 3? We can multiply w1 with v1, w2 with v2, w3 with v3, … wn with vn. Now we get a list of new vectors weighted_v1, weighted_v2, weighted_v3, … weighted_vn.</p><p><strong>Step 7: Sum up the weighted vectors. </strong>Sum up weighted_v1, weighted_v2, weighted_v3, … weighted_vn and then we get one vector z1 which represents “Robot1” with attention information added. By repeating steps 3 to 7, we can have z2, z3, … zn. Now, we have successfully transformed the original feature embedding in step 1 into vectors with attention in it.</p><p>And voila: self-attention in a nutshell.</p><p>In my project, the vectors discussed above can be seen as a new representation of each game feature — one that includes information on how related it is to all the other features.</p><p>Therefore, we have solved the problem of how to express the relationship between game resources in a way the computer can understand.</p><h3>In Conclusion</h3><p>Self-attention is the basis for understanding pre-training technologies such as <a href="https://medium.com/ai%C2%B3-theory-practice-business/attention-one-key-to-making-pre-training-successful-37b1c0b7f55a">Bert and XLnet</a>. With a good understanding, you can use it in more flexible ways, too, such as the recommendation system we created today. For any practitioner of AI, it is a key concept we can’t afford to ignore.</p><p>Next up, we’ll dive deeper into our discussion of pre-training technologies with another example from my work at <a href="https://www.yodo1.com/en/">Yodo1</a>. Don’t miss it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=794c796a3063" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai%C2%B3-theory-practice-business/attention-in-recommendation-systems-794c796a3063">Attention in Recommendation Systems</a> was originally published in <a href="https://medium.com/ai%C2%B3-theory-practice-business">AI³ | Theory, Practice, Business</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Attention: One Key to Making Pre-Training Successful]]></title>
            <link>https://medium.com/ai%C2%B3-theory-practice-business/attention-one-key-to-making-pre-training-successful-37b1c0b7f55a?source=rss-e25490321c4b------2</link>
            <guid isPermaLink="false">https://medium.com/p/37b1c0b7f55a</guid>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[attention]]></category>
            <dc:creator><![CDATA[dan lee]]></dc:creator>
            <pubDate>Mon, 30 Mar 2020 04:06:21 GMT</pubDate>
            <atom:updated>2020-04-01T06:44:17.086Z</atom:updated>
            <content:encoded><![CDATA[<h3>Attention in NLP: One Key to Making Pre-Training Successful</h3><h4>Pre-Training, Part 2: the mechanism that tells your model where to put its focus</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*NiOZ2qAWh6g2cYpweiTt1A.png" /></figure><p>On the job at Yodo1, one task of our AI team is to classify users into different groups. Recently, a model we used showed better performance with attention than without — enough to convince me that it’s a method worth trying, and worth sharing with you!</p><p>So today, we will be talking about the attention mechanism.</p><p><strong>What does the attention mechanism do?</strong></p><p>In the Encoder-Decoder model, the attention’s job is to focus on different elements of the encoder information based on what is most relevant at each step of the decoder process.</p><p>For a bit of background, the 2014 paper <a href="https://arxiv.org/abs/1409.0473">Neural Machine Translation by Jointly Learning to Align and Translate</a> is generally considered to be where the attention mechanism was originally put forward. It has only become more and more popular since.</p><p>Moving beyond this original attention — which we will refer to in this post as traditional attention — the mechanism has been developed into several variants, which mainly include:</p><ul><li>Soft vs hard attention;</li><li>Global vs local attention;</li><li>Self-attention.</li></ul><p>Most of these variants will not be addressed here: for now, it is enough simply to know that many exist. We will discuss self-attention in my next post as it is the basic idea behind the transformer which, in turn, is the basic framework used in <a href="https://medium.com/ai%C2%B3-theory-practice-business/what-is-pre-training-in-nlp-introducing-5-key-technologies-455c54933054">GPT and BERT</a>.</p><p>Both GPT and BERT, as introduced in <a href="https://medium.com/ai%C2%B3-theory-practice-business/what-is-pre-training-in-nlp-introducing-5-key-technologies-455c54933054">part one of this series</a>, are important models in pre-training. And as you may recall, GPT adopted the decoder part of the transformer while BERT adopted the encoder part.</p><p>Traditional attention is the basis of all of these ideas.</p><p>So, in this post, we’re going to learn more about the traditional attention mechanism, take a quick look at how it can be used, and dive into a deeper understanding with seq2seq attention.</p><p>Let’s make it happen!</p><h3>What Is The Attention Mechanism?</h3><p>To explain attention simply, we can use the task of translation as an example.</p><p>In the process of generating a translation in the target language, we focus on corresponding words or positions in the source language at each step.</p><p>In other words, when we do the translation, we don’t focus invariably on the entire source sentence. Instead, as the translation progresses, our focus travels step-by-step through the sentence, from one word to another.</p><p>The attention mechanism is designed to imitate this process. At different steps of a task, it adds the relevance of different pieces of previous information to the current step. This allows the model to highlight previous information that is most relevant to the current step and achieve better performance.</p><p>On the job at Yodo1, one task of our AI team is to classify users into different groups. Recently, a model we used showed better performance with attention than without — enough to convince me that it’s a method worth trying, and worth sharing with you today!</p><h3>A Quick Look: Using The Attention Layer In Tensowflow2.1</h3><p>Tensorflow2.1 recently provided us with an attention API: <em>tensorflow.keras.layers.Attention, </em>which we can use to quickly and directly build a model that contains the attention mechanism.</p><p>Let’s have a look at this CNN+ attention as an example:</p><pre># Variable-length int sequences.</pre><pre>query_input = tf.keras.Input(shape=(None,), dtype=&#39;int32&#39;)</pre><pre>value_input = tf.keras.Input(shape=(None,), dtype=&#39;int32&#39;)</pre><pre># Embedding lookup.</pre><pre>token_embedding = tf.keras.layers.Embedding(max_tokens, dimension)</pre><pre># Query embeddings of shape [batch_size, Tq, dimension].</pre><pre>query_embeddings = token_embedding(query_input)</pre><pre># Value embeddings of shape [batch_size, Tv, dimension].</pre><pre>value_embeddings = token_embedding(query_input)</pre><pre># CNN layer.</pre><pre>cnn_layer = tf.keras.layers.Conv1D(</pre><pre>filters=100,</pre><pre>kernel_size=4,</pre><pre># Use &#39;same&#39; padding so outputs have the same shape as inputs.</pre><pre>padding=&#39;same&#39;)</pre><pre># Query encoding of shape [batch_size, Tq, filters].</pre><pre>query_seq_encoding = cnn_layer(query_embeddings)</pre><pre># Value encoding of shape [batch_size, Tv, filters].</pre><pre>value_seq_encoding = cnn_layer(value_embeddings)</pre><pre># Query-value attention of shape [batch_size, Tq, filters].</pre><pre>query_value_attention_seq = tf.keras.layers.Attention()(</pre><pre>[query_seq_encoding, value_seq_encoding])</pre><pre># Reduce over the sequence axis to produce encodings of shape</pre><pre># [batch_size, filters].</pre><pre>query_encoding = tf.keras.layers.GlobalAveragePooling1D()(</pre><pre>query_seq_encoding)</pre><pre>query_value_attention = tf.keras.layers.GlobalAveragePooling1D()(</pre><pre>query_value_attention_seq)</pre><pre># Concatenate query and document encodings to produce a DNN input layer.</pre><pre>input_layer = tf.keras.layers.Concatenate()(</pre><pre>[query_encoding, query_value_attention])</pre><pre># Add DNN layers and create a Model.</pre><pre># ...</pre><p>Of course, you would have to make some modifications to the code based on your scenario if you really wanted your system to perform better. But this example should be enough to show how you can introduce attention to your model for a quick experiment.</p><h3>A Deeper Understanding: Seq2seq Attention</h3><h3>Background Knowledge Needed</h3><p>To understand this part, you need to have some background knowledge of:</p><ul><li>The Encoder-Decoder framework;</li><li>LSTM input and output.</li></ul><p>To understand the attention model in deep learning, you must be familiar with the Encoder-Decoder framework to which most of the current attention mechanisms are attached. At the same time, we should understand that the attention mechanism as a concept that does not depend on any particular framework.</p><p>The specific models used by Encoder and Decoder can be varying combinations of CNN, RNN, LSTM, GRU, etc. In machine translation models, the Encoder-Decoder framework used is often LSTM-LSTM.</p><p>We will also be using the LSTM-LSTM encoding-decoding framework in our second example, to help us gain a deeper understanding of the attention mechanism.</p><p>If you are not familiar with the above terms, I recommend spending a few minutes googling them. You don’t need a deep understanding; simply knowing what they look like will do for now.</p><h3>How Attention Works, Step by Step</h3><p>Let’s look at what we call the seq2seq attention model, where attention is used in the LSTM-LSTM encoding-decoding framework.</p><p>To make it easier to explain, we will use the simplest architecture — in which the encoder and decoder both have only one LSTM layer each.</p><p>Our task is to translate “C’est la vie!” into English, to show us step-by-step how seq2seq with an attention mechanism works.</p><p><strong>Step 1:</strong> We get the embedding vector of each word. Then we throw these embedding vectors into the LSTM layer and we get <em>h1, h2, h3</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*oHXWeiXXn2eSTVpi" /></figure><p><strong>Step 2:</strong> In the decoder, we input the embedding of <em>&lt;start&gt;</em> and <em>h3</em> into the LSTM to get <em>g1</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*FZ-7nJMNEZaRd2sX" /></figure><p><strong>Step 3:</strong> Now we need to get the attention vector, which is called context vector c. To do this, we must compute the dot product of <em>h1g1, h2g1, h3g1</em>.</p><p>The dot product can indicate how <em>g1</em> is relevant (similar) to<em> h1, h2, h3</em>.</p><p>Note: The dot product is not the only way to get this relevance. Two other popular methods are:</p><ul><li><em>hWg</em>, where <em>W </em>is the weight that can be trained;</li><li><em>vtanh(W(h;g))</em>, concatenating <em>h </em>and <em>g.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*PF7UvgSbrQOiKAkq" /></figure><p><strong>Step 4:</strong> Use softmax to normalize <em>h1g1, h2g1, h3g1,</em> and we get the score <em>alpha1, alpha2, alpha3.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*bui9gclXZv6YvbF9" /></figure><p><strong>Step 5: </strong>Now we compute <em>ci </em>= j=1Txijhj. In this example, <em>c1 = alpha1h1 + alpha2h2 + alpha3h3</em>. We can see that the context vector <em>c</em> is a weighted sum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*62Pr5ysx4tUnS6ms" /></figure><p><strong>Step 6.</strong> Now that we have <em>g1 </em>and <em>c1</em>, we can compute<em> y1: y1(before softmax) = f(c1, g1).</em></p><p>We concatenate <em>c1</em> and <em>g1</em> and input this into a function, which can be <em>W[c1;g1]</em> where <em>W </em>is the weight that can be trained. Then we input the result to softmax to get the distribution of words in the English dictionary and we can get the biggest probability corresponding word it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*zqAcU_Z8Hr3Dzlss" /></figure><p><strong>Result: </strong>By repeating steps 1 to 6 to get <em>g2, y2</em>, and so on, we will finally get the English translation: “It’s (the) life.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*EaNukig6jokh77Y5" /></figure><p>And voila! That’s a complete picture of the seq2seq attention model.</p><p>The details of the realization of attention variants can differ but the basic process will follow what we described above.</p><p>If you are interested in different kinds of attention, I highly encourage you to research the topic! And to learn about self-attention with me, you can come back next month.</p><h3>In Conclusion</h3><p>In the calculation process, attention will associate each piece of input information in the task with the current step by calculating its relevance. This greatly shortens the gap between long-distance dependent features, which is conducive to their effective use.</p><p>Next up, we’ll see how self-attention works. Don’t miss it!</p><p>Thanks for reading! If you enjoyed this article, please hit the clap button as many times as you can. It would mean a lot and encourage me to keep sharing my knowledge.</p><p>Feel free to share your questions and comments here and follow me for the latest content.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37b1c0b7f55a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai%C2%B3-theory-practice-business/attention-one-key-to-making-pre-training-successful-37b1c0b7f55a">Attention: One Key to Making Pre-Training Successful</a> was originally published in <a href="https://medium.com/ai%C2%B3-theory-practice-business">AI³ | Theory, Practice, Business</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Is Pre-Training in NLP? Introducing 5 Key Technologies]]></title>
            <link>https://medium.com/ai%C2%B3-theory-practice-business/what-is-pre-training-in-nlp-introducing-5-key-technologies-455c54933054?source=rss-e25490321c4b------2</link>
            <guid isPermaLink="false">https://medium.com/p/455c54933054</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[pre-training]]></category>
            <dc:creator><![CDATA[dan lee]]></dc:creator>
            <pubDate>Mon, 24 Feb 2020 07:18:21 GMT</pubDate>
            <atom:updated>2020-04-13T01:59:36.894Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Part 1: Building your understanding of pre-training AI</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*ETD7Fgs_3avjoMbhjvwO8A.png" /></figure><p>Welcome back to my blog for engineers who want to learn AI!</p><p>Starting with this post, we’ll be launching into a new series of articles on pre-training in NLP. Today, we’ll begin by forming a big picture.</p><p>In addition to defining pre-training, this article will give you a bird’s eye view of its development by discussing the main technologies at play. In the articles to follow, we’ll discuss each one in detail.</p><p>By the end of this post, you should understand what pre-training is and have a basic knowledge of the following key technologies:</p><ol><li>Word2vec</li><li>ELMo</li><li>GPT</li><li>BERT</li><li>XLNet</li></ol><h3>What is Pre-Training?</h3><p>Pre-training in AI<strong> </strong>refers<strong> </strong>to training a model with one task to help it form parameters that can be used in other tasks.</p><p>The concept of pre-training is inspired by human beings. Thanks to an innate ability, we don’t have to learn everything from scratch. Instead, we transfer and reuse our old knowledge of what we have learned in the past to understand new knowledge and handle a variety of new tasks.</p><p>In AI, pre-training imitates the way human beings process new knowledge. That is: using model parameters of tasks that have been learned before to initialize the model parameters of new tasks. In this way, the old knowledge helps new models successfully perform new tasks from old experience instead of from scratch.</p><h3>5 Essential Pre-Training Technologies</h3><p>Let’s build on our understanding of pre-training by briefly discussing five main methods, each of which will be addressed in further detail in later posts.</p><h3>1. Word2vec</h3><p><strong>What is Word2vec? </strong>Word2vec is a tool created by Google that produces static word embedding.</p><p>In 2013, Google open-sourced a famous tool for word embedding, word2vec, which can be efficiently trained on millions of words. It turns out that the word embedding trained by word2vec has the seemingly magical ability to measure word-to-word similarity.</p><p>So let’s take a quick look under the hood!</p><p>The word2vec uses is a shallow neural network with each word’s <em>one-hot embedding</em> as its input and output.</p><p><em>What does one-hot embedding look like?</em></p><p>If a dictionary has four words, {‘have’, ‘a’, ‘good’, ‘day’}, the one-hot embedding of the word “good” is [0, 0, 1, 0].</p><p>One way of training is to predict a target word (the word’s one-hot embedding) as output, with the one-hot embedding of its surrounding words as input. Another way is to predict the surrounding words as output with a target word as input, which corresponds individually to CBOW and Skip-gram.</p><p>After the training is finished, the parameter matrix of the model will have formed the word-embedding dictionary — so we get each word’s embedding of the training data.</p><p>In fact, when we talk about the word2vec algorithm or model, it actually refers to the CBOW and Skip-gram models. Many people think that word2vec itself refers to an algorithm or model but this is a misunderstanding.</p><p>On a final note, Hierarchical Softmax and Negative Sampling are optimization algorithms for training the CBOW and Skip-gram models.</p><h3>2. ELMo</h3><p><strong>What is ELMo? </strong>ELMo is an AI technology for dynamic word embedding with LSTM.</p><p>To understand ELMo, we must consider that the word embedding learned from Word2Vec is static: no matter the context, it will not change. This leads to a situation where word2vec’s word embedding can’t handle polysemous words.</p><p>But we can adjust the word embedding according to the context to give it a more semantic meaning — which is where ELMo comes into play. It is the idea of ​​dynamically adjusting word embedding according to the current context.</p><p>ELMo’s network architecture uses a two-layer bidirectional LSTM on top of the word embedding layer learned by word2vec.</p><p>To understand how this works, let’s work up from the word2vec layer, the product of which is embedding1.</p><p>Next, we add two bidirectional LSTM layers, where each layer is composed of two LSTMs — one from left to right and the other one from right to left. This gives us a second and third layer on top of the first. If we take a closer look at layer2, as an example, we’ll see that the two LSTMs within it generate two new embeddings.</p><p>After concatenating the two embeddings, we have a new <em>embedding2 </em>of this layer, which we feed into the third layer. The bidirectional LSTM in layer3 generates <em>embedding3 </em>in the same way as in layer2.</p><p>Then all three embeddings are weighted and summed up. Now, a new word embedding is obtained with syntactic (<em>embedding2, </em>the second-layer output) and semantic (<em>embedding3, </em>the third-layer output) information coded in it.</p><p>ELMo’s pre-training process not only learns word embedding but also learns a two-layer, bi-directional LSTM network structure — both of which have their own roles when facing new tasks.</p><h3>3. GPT</h3><p><strong>What is GPT? </strong>In AI, GPT is a transformer-decoder-based autoregressive language model.</p><p>The core idea behind the Transformer model is self-attention — the ability to attend to different positions of the input sequence to compute a representation of that sequence. To understand the Transformer, we need to discuss the attention mechanism and self-attention mechanism — which we will in detail in future articles.</p><p>You can also refer to Jay Alammar’s blog post, <a href="http://jalammar.github.io/illustrated-transformer/">The Illustrated Transformer</a>, which gives a very clear explanation of the Transformer. But for now, we only need to understand that the Transformer consists of two parts: an encoder and a decoder.</p><p>GPT is a method based on the Transformer <em>decoder</em>. GPT also uses static word embedding as input, plus a certain number of layers of the Transformer decoder on top of it.</p><p>As an example, let’s take another four-word sentence: “w1, w2, w3, w4.”</p><p>Now let’s take w3 as the current word. After its embedding passes through a decoder layer, it will become a new embedding — which will include the attention information w3 paid to w1 and w2. Note that w1 and w2 are to the left of w3.</p><p>We will explain attention information in articles to come but for now, you can simply think of it as a new kind of embedding. This gives the new embedding a better representation to better complete the task of predicting the next word from left to right.</p><p>When fine-tuning specific NLP tasks with supervised learning, GPT — unlike ELMo, which connects to other model layers as feature representations — does not need to re-build new model structures for tasks. Instead, it directly connects the last layer to softmax as the task output layer, then fine-tunes the entire model.</p><h3>4. BERT</h3><p><strong>What is BERT? </strong>In AI, BERT is a Transformer-encoder-based autoencoder language model.</p><p>In the paper “<a href="https://arxiv.org/abs/1810.04805">BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</a>,” Google divided pre-trained language representation methods into feature-based methods (ELMo) and fine-tuning-based methods (GPT).</p><p>BERT is a fine-tuning-based and <em>encoder-</em>based method (where you wil recall that GPT is decoder-based).</p><p>BERT changes the unidirectional language model in GPT into a bidirectional one. Instead of using the standard left-to-right prediction of the next word as the target task, BERT proposes two new tasks.</p><p>BERT’s first pre-training task is called MLM, or Masked Language Model. In the input word sequence of this model, 15% of the words are randomly masked and the task is to predict what they are. What we see is that, unlike previous models, BERT can predict these words from both directions — not just left-to-right or right-to-left.</p><p>In addition, in terms of model input, BERT WordPiece embedding is used instead of standard word embedding. That’s because BERT, by default, doesn’t use words as tokens but word pieces. For example, ‘playing’ is ‘play’, ‘ing’ instead of ‘playing’. From here, a position embedding and a segment embedding are added.</p><p>The position embedding alleviates the shortcomings of self-attention in ignoring word position information. The segmentation embedding represents the division of two sentences. The two input sentences are represented by 0 and 1 in the segment embedding.</p><p>After summing up the WordPiece embedding, position embedding, and segment embedding, we get a new embedding with word information and position information. Then we input this embedding to the BERT Transformer encoder layer to start the training.</p><h3>5. XLNet</h3><p><strong>XLNet: </strong>XLNet is an AI technology that integrates GPT and BERT</p><p>XLNet introduces us to the <em>autoregressive</em> model and DAE (denoise <em>autoencoder) </em>model.</p><p>The language model (LM) that predicts the next possible word based on the <em>before-context </em>(words to the left) or predicting the previous word based on the <em>after-context</em> (words to the right) is called the autoregressive LM.</p><p>Autoregressive language models have naturally matched tasks to generate the next words. The autoencoder model, on the other hand, can naturally integrate into the bidirectional language model. This makes it excellent for tasks that require both before-context and after-context, such as reading comprehension.</p><p>The idea behind XLNet is: Can we take advantage of both autoregressive LM and DAE LM?</p><p>That is, XLNet tries to introduce the bidirectional language model into the autoregressive language model.</p><p>We know that the autoregressive language model reads from left to right, meaning the model can’t see the <em>after-context — </em>what is to the right of the current word. Knowing how we make the model see words to the right of the current word with an autoregressive LM, which can only see words on the left, is equivalent to know how to implement a bidirectional language model in the autoregressive language model.</p><p>XLNet’s approach is to fix the current word position in the sentence. That is, if the current word, <em>w, </em>is at position 3, then it stays in position 3 but the other words in the sentence are randomly shuffled to different positions. This way, even if the model can only read what is to the right, it still has a way to see the other words in the sentence.</p><p>We can get a list of possible sentences, and then randomly choose some sentences among the possibilities as input for model pre-training. In this way, we see both the words on the left and those to the right of the current word, and the form of the autoregressive model remains unchanged.</p><p>Of course, in practice, the above ideas are not implemented in the input step but in other processes. But that’s a story for another time and something we’ll address in a future XLNet article.</p><h3>In Conclusion</h3><p>Pre-Training is the most fruitful branch of current NLP research. Arguably, it changed the research way of NLP in the past. Understanding Pre-Training is an essential process for learning AI.</p><p>My next article would be:</p><p><a href="https://medium.com/ai%C2%B3-theory-practice-business/attention-one-key-to-making-pre-training-successful-37b1c0b7f55a">Attention in NLP: One Key to Making Pre-Training Successful</a></p><p>If you are interested in this article and want to discuss this topic more deeply, please feel free to leave comments or contact me at gglfxsld@gmail.com.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=455c54933054" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai%C2%B3-theory-practice-business/what-is-pre-training-in-nlp-introducing-5-key-technologies-455c54933054">What Is Pre-Training in NLP? Introducing 5 Key Technologies</a> was originally published in <a href="https://medium.com/ai%C2%B3-theory-practice-business">AI³ | Theory, Practice, Business</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reinforcement Learning, Part 7: A Brief Introduction to Deep Q Networks]]></title>
            <link>https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-7-a-brief-introduction-to-deep-q-networks-aa45314a2ae?source=rss-e25490321c4b------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa45314a2ae</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <dc:creator><![CDATA[dan lee]]></dc:creator>
            <pubDate>Thu, 30 Jan 2020 14:39:39 GMT</pubDate>
            <atom:updated>2020-02-24T07:23:52.253Z</atom:updated>
            <content:encoded><![CDATA[<h4>Build your understanding of DQN in three easy steps</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*9ROcRFWJxEowwv9Y.jpg" /></figure><p>Welcome back to my series on Reinforcement Learning! Now that we’ve covered <a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-6-td-%CE%BB-q-learning-99cdfdf4e76a">Q-learning</a>, it’s time to introduce Deep Learning within RL.</p><p>In this post, we’ll discuss:</p><ul><li>Why the introduction of Deep Learning to RL is so important;</li><li>What challenges it comes with and how we can solve them; and how can we solve them;</li><li>How we can build a DQN.</li></ul><p>If you’re new to the series, be sure to check out these posts first:</p><p><a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-1-a-brief-introduction-a53a849771cf">Part 1: A Brief Introduction to RL</a></p><p><a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-2-introducing-markov-process-d3586d4003e0">Part 2: Introducing the Markov Process</a></p><p><a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-3-the-markov-decision-process-9f5066e073a2">Part 3: Markov Decision Process (MDP)</a></p><p><a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-4-optimal-policy-search-with-mdp-7fc96158ea8a">Part 4: Optimal Policy Search with MDP</a></p><p><a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-5-monte-carlo-and-temporal-difference-learning-889053aba07d">Part 5: Monte-Carlo and Temporal-Difference Learning</a></p><p><a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-6-td-%CE%BB-q-learning-99cdfdf4e76a?source=your_stories_page---------------------------">Part 6:TD(λ) &amp; Q-learning</a></p><p>Now it’s time to dive in a little deeper!</p><h3>Why Deep Learning?</h3><p>In Q-learning, when the state and action space are discrete and the dimension is low, a Q-Table can be used to store the Q value of each state-action pair. However, when the state and action space are high-dimensional and continuous, a Q-Table wouldn’t be qualified.</p><p>A common solution is to turn the Q-Table-update problem into a Function-fitting problem and get similar output actions for similar states. The Q function approximates the optimal Q value by updating the parameter θ:</p><p>Q (s, a; θ) ≈Q ′ (s, a)</p><p><strong>Deep neural networks</strong>, which<strong> </strong>can automatically extract complex features. They are the ticket for dealing with high-dimensional Q-Tables with continuous states.</p><p><strong>DQN </strong>is one of many algorithms that combines Deep Learning and Reinforcement Learning to learn strategies directly from high-dimensional raw data. By combining a convolutional neural network (CNN) with Q-Learning, DQN promoted the development of Reinforcement Learning and expanded its application scenarios.</p><p>The input of CNN can be the original image data (as the state), and the output is each action-corresponding value evaluation (Q value).</p><p>Google’s Deepmind team published two papers on DQN:</p><ul><li><a href="https://arxiv.org/pdf/1312.5602.pdf">Playing Atari with Deep Reinforcement Learning</a> on NIPS in 2013;</li><li><a href="http://www.davidqiu.com:8888/research/nature14236.pdf">Human-level control through deep reinforcement learning</a> on Nature in 2015.</li></ul><p>As these articles cover the foundational idea of DQN, I’ll reference them throughout this post.</p><h3>Step 1: Understanding the Challenges of Applying Deep Learning to RL</h3><h3>Distribution of Samples</h3><p>A major reason for deep learning to converge with RL is that data sets require independent and identical distributions. Only models trained by deep learning on this kind of data set could better fit the potential patterns in the data set and obtain ideal results.</p><p>But, as we’ve discussed extensively in this series, Reinforcement Learning learns from reward and a series of highly related states. The agents may change the data distribution as the algorithm learns new behaviors, leading to the inability to converge.</p><p>Therefore, if you want to apply deep learning algorithms to Reinforcement Learning, you must cut off the correlation of states and stabilize the distribution of the data set.</p><h3>Supervised Issues</h3><p>Deep learning is a standard form of supervised learning. Before training, the target of the ideal model that you want to train already exists — only it’s hidden. Through deep iteration, deep learning makes the algorithm converge so that it can find the ideal model.</p><p>In contrast, Reinforcement Learning is learning from sparse, noisy, and delayed reward signals. If you want to apply deep learning algorithms to Reinforcement Learning, you must design a target for each update iteration.</p><p>For a refresher on the difference between supervised and unsupervised learning, jump back to my first post: <a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-1-a-brief-introduction-a53a849771cf">a brief introduction to Reinforcement Learning</a>.</p><h3>Step 2: Solving The Contradiction Between Supervised Deep Learning And Unsupervised RL</h3><p>Here are two solutions to the challenges discussed above.</p><h3>Sample Distribution</h3><p>According to <a href="https://arxiv.org/pdf/1312.5602.pdf">Deepmind’s 2013 paper</a>, the experience mechanism can solve the problem of the sample distribution. Here’s how it works:</p><ol><li><strong>Initialize. </strong>The agent will be initialized with a state <em>s</em>, then input the state<em> s</em> into the evaluation network (a neural network, described later), and output the <em>q</em> value of each action.</li><li><strong>Select action. </strong>The agent selects the action performed in this state through the <em>q</em> value and the exploration and exploitation algorithm.</li><li><strong>Feedback. </strong>At the state <em>s, </em>we choose action <em>a,</em> the environment will provide feedback — in the form of a reward — and give the agent its next state <em>s’</em>.</li><li><strong>Sequence. </strong>At this point, a sequence (<em>s, a, r, s’</em>) is reached, where <em>s</em> is the current state, <em>a</em> is the action performed in state <em>s</em>, <em>r</em> is the reward that the environment feeds back to the agent, and<em> s’ </em>is the next state. A sequence (<em>s, a, r, s’</em>) constitutes a training sample.</li><li><strong>Parameters. </strong>Set the size of the experience pool to <em>N</em>. That means, only <em>N</em> latest sequences exist in the experience pool. Sequences larger than <em>N</em> will overwrite the samples in the experience pool. The data in the empirical pool is used every time the neural network parameters are updated.</li><li><strong>Convergence. </strong>The existence of the experience pool turns a series of highly correlated states into discrete data samples. It can reduce the training variance and stabilize the sample distribution, which is conducive to the convergence of the algorithm. Just like a human being, an agent can learn from its own historical experience.</li></ol><h3>Solving Supervised Issues</h3><p>In order to solve the contradiction between Reinforcement Learning and supervised deep learning, <a href="https://arxiv.org/pdf/1312.5602.pdf">Deepmind’s 2013 paper</a> outlines the designs of two neural networks.</p><p>One neural network is a prediction network (<strong>critic</strong>):</p><ul><li>The input is the current state — the first element <em>s</em> in the sample sequence. The output is the <em>q</em> value generated by each action.</li><li>The agent uses the <em>q</em> value of each output to determine the state to be executed.</li><li>The parameters of this network can be updated at any time.</li></ul><p>The other neural network is the evaluation network (<strong>actor</strong>):</p><ul><li>The input is the next state; that is, the fourth element <em>s’</em> in the sample sequence. The output is the <em>q</em> value <em>(q_next) </em>of each action of the state <em>s’</em>.</li><li>Next, the Bellman equation <em>(q_target = r + γ * max (q_next))</em> calculates the target <em>q</em> value for performing action <em>a</em> in the state <em>s</em>.</li><li>The output of the evaluation network passes through the Bellman equation, and the target <em>q</em> value of the calculated result can be used as the label of the prediction network.</li></ul><p>Thus, the above [steps/networks] can solve the contradiction between supervised deep learning and unsupervised Reinforcement Learning.</p><p>In order to suppress the problem of correlation between states, the parameters of the evaluation network are not updated in real-time. Instead, after a certain number of steps, the prediction network will copy its parameters to the evaluation network.</p><p>Now, let’s put the main architecture of DQN into simple terms.</p><h3>Step 3: Building Your Own DQN</h3><h3>Replay Memory</h3><p>Replay memory stores the experiences that the <strong>actor </strong>DQN records via playing the game. These experiences will be used as training data by the <strong>critic </strong>DQN later on.</p><p>By sampling randomly from the experience data, a batch of transitions is built and data in it are decorrelated.</p><h3>How Does DQN Work?</h3><p>The <strong>actor </strong>DQN can be used to play games, though initially, it does so very badly.</p><p>Let the actor play for a while, then save all its experiences in a <em>replay memory</em>. Each memory will be a<em> 5-tuple</em> (state, action, next state, reward, continue).</p><p>The “continue” item will be 0.0 when the game is over or 1.0 otherwise.</p><p>At regular intervals, we can sample a batch of memories from the replay memory — just like we do with dynamic training data in supervised learning. We use these memories to estimate the Q-Values.</p><p>You want the actor to explore the game thoroughly, so you may want to combine it with a ε-greedy policy.</p><p>The <strong>critic </strong>DQN will adjust its parameters to make its Q-Value predictions approximate the result estimated by the actor through the actor’s experience data. Regular supervised learning techniques can work on those data.</p><p>Copy the critic DQN to the actor DQN at regular intervals, and that’s it!</p><h3>Summary</h3><p>In short, DQN is a success! And you’re well on your way to understanding and being able to use it!</p><p>You should now be familiar with basic concepts of:</p><ul><li><strong>Deep Reinforcement Learning (DRL):</strong> the combination of DL and RL</li><li><strong>DQN: </strong>a typical method of DRL</li></ul><p>In my next post, we will explore DRL even more deeply.</p><p><strong>Thanks for reading! If you enjoyed this article, please hit the clap button as many times as you can. It would mean a lot and encourage me to keep sharing my knowledge.</strong></p><p><em>Questions or thoughts to add? I’d love to hear from you in the comments!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa45314a2ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-7-a-brief-introduction-to-deep-q-networks-aa45314a2ae">Reinforcement Learning, Part 7: A Brief Introduction to Deep Q Networks</a> was originally published in <a href="https://medium.com/ai%C2%B3-theory-practice-business">AI³ | Theory, Practice, Business</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reinforcement Learning, Part 6: TD(λ) & Q-learning]]></title>
            <link>https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-6-td-%CE%BB-q-learning-99cdfdf4e76a?source=rss-e25490321c4b------2</link>
            <guid isPermaLink="false">https://medium.com/p/99cdfdf4e76a</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[q-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <dc:creator><![CDATA[dan lee]]></dc:creator>
            <pubDate>Fri, 13 Dec 2019 10:57:43 GMT</pubDate>
            <atom:updated>2019-12-14T10:35:32.012Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Understanding Q-learning in action and by example</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ecmY_SV1ajZsUM-a2GF_hQ.png" /></figure><p>Welcome back to my series on Reinforcement Learning! Now that we’ve covered the building blocks, it’s time to discuss TD(λ) and Q-learning.</p><p>In this post, I’ll use a simple example to help you understand Q-learning and answer the following questions:</p><ul><li>What is <strong>TD(λ) </strong>and how is it used?</li><li>How does <strong>Q-learning</strong>’s classic off-policy<strong> </strong>method work?</li><li>What does a <strong>Python</strong> realization of Q-learning look like?</li></ul><p>If you’re new to the series, be sure to check out these posts first:</p><p><a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-1-a-brief-introduction-a53a849771cf">Part 1: A Brief Introduction to RL</a></p><p><a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-2-introducing-markov-process-d3586d4003e0">Part 2: Introducing the Markov Process</a></p><p><a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-3-the-markov-decision-process-9f5066e073a2">Part 3: Markov Decision Process (MDP)</a></p><p><a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-4-optimal-policy-search-with-mdp-7fc96158ea8a">Part 4: Optimal Policy Search with MDP</a></p><p><a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-5-monte-carlo-and-temporal-difference-learning-889053aba07d">Part 5: Monte-Carlo and Temporal-Difference Learning</a></p><p>Let’s get started with Q-learning!</p><h3>Temporal-Difference Learning: TD(λ)</h3><p>In my <a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-5-monte-carlo-and-temporal-difference-learning-889053aba07d">last post</a>, we mentioned that if<em> </em>we replace<em> Gt </em>in the MC-updated formula with an estimated return<em> Rt+1+V(St+1), </em>we can get <em>TD(0)</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rFFH-32xt4-7JPuPm0BhXw.png" /></figure><p>Where:</p><ul><li><em>Rt+1+V(St+1) </em>is called <strong>TD target value</strong></li><li><em>Rt+1+V(St+1)- V(St）</em>is called <strong>TD error</strong></li></ul><p>Now, we replace the <strong>TD target value </strong>with<strong> Gt(), </strong>we can have TD(λ). <strong>Gt()</strong>is generated as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IJcZPY8K51V2ERSVGOYOEQ.png" /></figure><p>So the <em>TD(λ)</em> formula is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RzD7A7K_K34IFLPYNHvfWw.png" /></figure><p>In which:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/0*BRQdbfMW5oTlATw6" /></figure><p>As discussed, Q-learning is a combination of Monte Carlo (MC) and Temporal Difference (TD) learning. With MC and <em>TD(0) </em>covered in <a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-5-monte-carlo-and-temporal-difference-learning-889053aba07d">Part 5</a> and <em>TD(λ) </em>now under our belts, we’re finally ready to pull out the big guns!</p><h3>Q-Learning</h3><p>Q-Value formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b2sPeJxXRqo9hBz9nnWE6w.png" /></figure><p>From the above, we can see that Q-learning is directly derived from <em>TD(0)</em>. For each updated step, Q-learning adopts a greedy method: maxaQ (St+1, a).</p><p>This is the main difference between Q-learning and another TD-based method called Sarsa, which I won’t explain in this series. But as an RL learner, you should know that Q-learning is not the only method based on TD.</p><h3>An Example of How Q-Learning Works</h3><p>Let’s try to understand this better with an example:</p><p>You’re having dinner with friends at an Italian restaurant and, because you’ve been here once or twice before, they want you to order. From experience, you know that the Margherita pizza and pasta Bolognese are delicious. So if you have to order ten dishes, experience might tell you to order five of each. But what about everything else on the menu?</p><p>In this scenario, you are like our “agent”, tasked with finding the best combination of ten dishes. Imagine this becomes a weekly dinner; you’d probably start bringing a notebook to record information about each dish. In Q-learning, the agent collects <strong>Q-values </strong>in a <strong>Q-table</strong>. For the restaurant menu, you could think of these values as a score for each dish.</p><p>Now let’s say your party is back at the restaurant for the third time. You’ve got a bit of information in your notebook now but you certainly haven’t explored the whole menu yet. How do you decide how many dishes to order from your notes — which you know are good, and how many new ones to try?</p><p>This is where <strong>ε-greedy </strong>comes into play.</p><h3>The ε-greedy Exploration Policy</h3><p>In the above example, what happened in the restaurant is like our MDP (<a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-3-the-markov-decision-process-9f5066e073a2">Markov Decision Process)</a> and you, as our “agent” can only succeed in finding the best combination of dishes for your party if you explore it thoroughly enough.</p><p>So it is with Q-Learning: it can work only if the agent explores the MDP thoroughly enough. Of course, this would take an extremely long time. Can you imagine how many times you’d have to go back to the restaurant to try every dish on the menu in every combination?</p><p>This is why Q-learning uses the <strong>ε-greedy policy</strong>, which is ε degree “greedy” for the highest Q values and <em>1 — </em>ε degree “greedy” for random exploration.</p><p>In the initial stages of training an agent, a random exploration environment (i.e. trying new things on the menu) is often better than a fixed behavior mode (i.e. ordering what you already know is good) because this is when the agent accumulates experience and fills up the Q-table.</p><p>Thus, it’s common to start with a high value for ε, such as 1.0. This means the agent will spend 100% of its time exploring (e.g. using a random policy) instead of referring to the Q-table.</p><p>From there, the value of ε can be gradually decreased, making the agent more greedy for Q-values. For example, if we drop ε to 0.9, it means the agent will spend 90% of its time choosing the best strategy based on Q-table, and 10% of its time exploring the unknown.</p><p>The advantage of the <em>ε</em>-greedy policy, compared to a completely greedy one, is that it always keeps testing unknown regions of the MDP. Even when the target policy seems optimal, the algorithm never stops exploring: it just keeps getting better and better.</p><p>There are various functions for exploration, and many defined exploration policies can be found online. Do note that not all exploration policies are expected to work for both discrete and continuous action spaces.</p><h3>A Python Realization of Q-Learning</h3><p>Here is how Q-learning can be implemented:</p><pre>import numpy as np</pre><pre># Q</pre><pre>q = np.matrix(np.zeros([6, 6]))</pre><pre># Reward</pre><pre>r = np.matrix([[-1, -1, -1, -1,  0,  -1],</pre><pre>[-1, -1, -1,  0, -1, 100],</pre><pre>[-1, -1, -1,  0, -1,  -1],</pre><pre>[-1,  0,  0, -1,  0,  -1],</pre><pre>[ 0, -1, -1,  0, -1, 100],</pre><pre>[-1,  0, -1, -1,  0, 100]])</pre><pre>gamma = 0.8</pre><pre>epsilon = 0.4</pre><pre># the main training loop</pre><pre>for episode in range(101):</pre><pre># random initial state</pre><pre>state = np.random.randint(0, 6)</pre><pre># if not final state</pre><pre>while (state != 5):</pre><pre># choose a possible action</pre><pre># Even in random case, we cannot choose actions whose r[state, action] = -1.</pre><pre>possible_actions = []</pre><pre>possible_q = []</pre><pre>for action in range(6):</pre><pre>if r[state, action] &gt;= 0:</pre><pre>possible_actions.append(action)</pre><pre>possible_q.append(q[state, action])</pre><pre># Step next state, here we use epsilon-greedy algorithm.</pre><pre>action = -1</pre><pre>if np.random.random() &lt; epsilon:</pre><pre># choose random action</pre><pre>action = possible_actions[np.random.randint(0, len(possible_actions))]</pre><pre>else:</pre><pre># greedy</pre><pre>action = possible_actions[np.argmax(possible_q)]</pre><pre># Update Q value</pre><pre>q[state, action] = r[state, action] + gamma * q[action].max()</pre><pre># Go to the next state</pre><pre>state = action</pre><pre># Display training progress</pre><pre>if episode % 10 == 0:</pre><pre>print(&quot;------------------------------------------------&quot;)</pre><pre>print(&quot;Training episode: %d&quot; % episode)</pre><pre>print(q)</pre><p>Given enough iterations, this algorithm will converge to the optimal Q-Values. This is called an <strong>off-policy</strong><em> </em><strong>algorithm </strong>because the policy being trained is not the one being executed.</p><p>Note: The code above is from <a href="https://blog.csdn.net/zjm750617105/article/details/80295267">https://blog.csdn.net/zjm750617105/article/details/80295267</a></p><h3>In Summary</h3><p>Q-Learning is an off-policy algorithm based on the TD method. Over time, it creates a Q-table, which is used to arrive at an optimal policy. In order to learn that policy, the agent must explore. The usual way to do this is by making the agent follow a different, random policy that initially ignores the Q-table when selecting actions.</p><p>And that’s all for today; you made it! With this article, we’ve rounded off our discussion of TD and from it, I hope you‘ve gained a basic knowledge of:</p><ul><li>Temporal-Difference Learning: TD target value, TD error, and TD(y)</li><li>A Python realization of Q-learning</li><li>Q-learning exploration policy with <em>ε-greedy</em></li></ul><p>TD and Q-learning are quite important in RL because a lot of optimized methods are derived from them. There’s Double Q-Learning, Deep Q-Learning, and so the list goes on.</p><p>On top of that, there are some other methods — quite different from TD and Q-learning — which we haven’t touched on in this series yet, like Policy Gradient (PG).</p><p>I will be introducing many of these methods in upcoming posts. If there’s anything you’re particularly eager to learn, please feel free to leave me a message in the comments. I’ll determine the order of priority based on your needs.</p><p><strong>Thanks for reading! If you enjoyed this article, please hit the clap button as many times as you can. It would mean a lot and encourage me to keep sharing my knowledge.</strong></p><p><em>Questions or thoughts to add? I’d love to hear from you in the comments!</em></p><p><em>Follow me here to catch my next post.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99cdfdf4e76a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-6-td-%CE%BB-q-learning-99cdfdf4e76a">Reinforcement Learning, Part 6: TD(λ) &amp; Q-learning</a> was originally published in <a href="https://medium.com/ai%C2%B3-theory-practice-business">AI³ | Theory, Practice, Business</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>