Design Professional Mails using Flask!

Have you ever wondered how companies like Medium send Emails to users that look so fancy that it is basically a website in a mail? And on top of that — they send millions of Emails to their Massive user-base in minutes DAILY!!! All Customized to the user's interests.

and if you don’t know JUST OPEN ANY OF YOUR EMAIL TAGGED PROMOTIONS

They have Images, Subscribe Buttons, Tables, GIF’s, Booking Buttons, Headers (in the center), Logos, Amazing Fonts, Socials, and god knows what else.

So let’s make our own “Automatic-Pro-PythonCode-Email-Sender-Bot…ish”

Let’s write a few lines of Python code that will allow us to sent emails to 1000’s of people at once just by running that python script.

Lets pip First

We are going to use Flask and Flask-mail to do this. So open your command prompt & do.

pip install Flask Flask-Mail

Code 💻

You only need 3 easy steps for this:

1. Create a Flask App

Let's start by creating a Flask app.

We will be using mail variable to manage the features the instance “Mail” present in the Flask-Mail library.

2. Add Your Details

3. Compose & Send [Simple]

🍓 Compose & Send [Advanced]

By using .html instead of .body you can use different html tags and add .svg, .png too. I am using <img> argument to add image using Url.

🍓 Compose & Send [Pro]

Now that we know how add text & image. We can use different .html tags in combination with text and images to make it impressive like any other company Email. Also add <button> </button> redirecting users to your new websites.

Book Tickets 🎫 button redirects you to the website www.bookmyshow.com by opening a new window.

Get the complete code on GitHub and try it yourself.

Sagarsharma4244/FlaskMail

💁 Keep in Mind

Since our code is not a full fledged app at this point, you may need to toggle your Gmail Setting.

Turn it ON!

How can I Send Personalized Email to 1000’s of Users?!

Automate Mails 📬

(like) Change User’s Name — Use python to go through a .txt or excel document of your user base and parse through all the names one by one and replace it in the mail.

!dea: You can also change all the “he” to “she” or vice-versa depending on the gender.

How can I Attach Files with the Mail ?!

Attaching Files 📎

Here we are attaching image.png file present in the same folder. You can also add pdf file by changing the name and extension.

Any comments or if you have any question, write it in the comment section.

Clap it! Share it! Follow Me !

Happy to be helpful. kudos…

Previous stories you will love:

• Activation Functions in Neural Networks
• Neural Networks on Mobile Devices with TensorFlow Lite: A Tutorial
• TensorFlow Image Recognition Python API Tutorial

How to Send Emails using Python was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

This will be a practical, end-to-end guide on how to build a mobile application using TensorFlow Lite that classifies images from a dataset for your projects.

This application uses live camera and classifies objects instantly. The TFLite application will be smaller, faster, and more accurate than an application made using TensorFlow Mobile, because TFLite is made specifically to run neural nets on mobile platforms.

We’ll be using the MobileNet model to train our network, which will keep the app smaller.

Getting Started

Requirements…

1. Python 3 .5 or higher— python3 -V
2. Tensorflow 1.9 or higher — pip3 install — upgrade tensorflow

Also, open the terminal and type:

alias python=python3

Now, python3 will open with the python command. This will make it easier to implement the code just by copy-pasting without having to worry about 3 after typing Python.

The TFLite tutorial contains the following steps:

Step 1: Download the Code Files

Let’s start by downloading the code from the tensorflow-for-poets GitHub. Open the command prompt where you want to download the folder and type:

This will download the files and make a new folder called tensorflow-for-poets in your current directory.

Output

FYI: You can change the name of the folder to your project name after downloading.

Info: The folder contains the sub-folders

scripts — Contains the machine learning code .py files.

tf_files — It will contain output files like models — graph.pb, labels.txt

android — Contains Android app projects for both tfmobile and TFlite.

iOS — Contains the iOS app project files using xCode.

Step 2: Download the Dataset

Let’s download a 200MB publicly available dataset with 5 different flowers to classify from. Then extract the flower_photos.tgz inside the tf_files folder which will look something like this:

tensorflow-for-poets-2 > tf_files > flower_photos

Info — The 5 different category folders are Rose, Daisy, Dandelion, Sunflower and Tulip.

Step 3: Retrain the model

Open the command prompt inside the “tensorflow-for-poets-2” folder and type:

Output

This will download the pre-trained frozen graph mobilenet_1.0_244

and create retrained_graph.pb and retrained_labels.txt files in the tf_files folder.

Open TensorBoard

Open another command prompt in the current directory and point TensorBoard to the summaries_dir:

Output

Now you can open the 6006 port in your browser to see the results.

Visualization

🔴 Training & 🔵 Validation

Accuracy is above 0.90 and loss is below 0.4 for validation.

Verify ✔️

I downloaded a random rose image from the Internet in the current folder and used the following command to run the label_image.py script to detect the --image using the --graph file.

Output

The result gives 99% accuracy for the new_rose image.

Step 4: Convert the Model to TFLite Format

TOCO (TensorFlow Lite Converter) is used to convert the file to .lite format. For more detail on TOCO arguments, use toco --help

This will create an optimized_graph.lite file in your tf_files directory.

Info: Use the toco -h for more details

--input_file has been updated to --graph_def_file

--input_format is not need for the mobile_net graphs.

From here on, the tutorial is divided into two sections: iOS and Android.

🍎 iOS 📱

Step 5: Setup Xcode Studio and Test Run

Download Xcode

Sign in with your Apple ID - Apple Developer

Install Xcode

Install Cocoapods

Install TFLite Cocoapod

Open the project with Xcode

Test Run 🏃

Press ▶️ to initiate the simulator in Xcode.

Step 6: Run your Application

First move the trained files into the assets folder of the application.

Replace the graph.lite file.

And then thelabels.txt file.

Now just click the ▶️ to open the simulator and drop images to see the results.

👏 👏 👏 Congratulations! Now you can apply the same method in your next gazillion dollar app, enable doctors work to faster and better without expensive equipment in the rural parts of the world, or just have fun. 👏👏 👏

🍭 🍦 Android 🍞🐝 🍩

Step 5: Setup Android Studio and Test Run

There are two ways to do this: Android Studio and Bazel. I’ll be using AS since more people are familiar with it.

If you don’t have it installed already, go here and install it:

Download Android Studio & App Tools - Android Developers

Test Run 🏃

To make sure everything is working correctly in Android Studio, let’s do a test run.

🔸 Open Android Studio and select “📁Open an existing Android Studio project”.

🔸 Go to android/tfmobile directory.

🔸 If everything works perfectly, click the BUILD>BUILD APK button.

Open the folder containing app-debug.apk file by clicking locate.

Note: Turn On the developer mode in your phone before installing the app.

Step 6: Run your Application

First move the trained files into the assets folder of the application.

Replace the graph.lite file.

and then labels.txt file.

Now click Tools>> Build .apk file.

Install Application

Install the .apk file onyour phone and see the re-trained neural network detecting the objects.

👏 👏 👏 Congratulations! Now you can apply the same method in your next gazillion dollar app, enable doctors to work faster and better without expensive equipment in rural parts of the world, or just have fun. 👏👏 👏

If you hit a wall while implementing this post, reach out in the comments.

For further tutorials on how to use TensorFlow in mobile apps follow me on Medium and Twitter to see similar posts.

Clap it! Share it! Follow Me!

Happy to be helpful. kudos…

Discuss this post on Hacker News and Reddit.

Editor’s Note: Heartbeat is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.

Editorially independent, Heartbeat is sponsored and published by Comet, an MLOps platform that enables data scientists & ML teams to track, compare, explain, & optimize their experiments. We pay our contributors, and we don’t sell ads.

If you’d like to contribute, head on over to our call for contributors. You can also sign up to receive our weekly newsletters (Deep Learning Weekly and the Comet Newsletter), join us on Slack, and follow Comet on Twitter and LinkedIn for resources, events, and much more that will help you build better ML models, faster.

Neural Networks on Mobile Devices with TensorFlow Lite: A Tutorial was originally published in Heartbeat on Medium, where people are continuing the conversation by highlighting and responding to this story.

They are also known as policy iteration & value iteration since they are calculated many times making it an iterative process.

Let’s understand why are they so important in Machine Learning and what’s the difference between them?

What is a Policy Network?

Consider any game in the world, input 🎮 given by user to the game is known as actions a. Every input (action) leads to a different output. These output are known as states s of the game.

From this, we can make different state-action pairs S = {(s0,a0),s1,a1),...,(sN,aN)} , representing which actions aN leads to which states sN. Also, we can say that S contains all the policies learned by the policy network.

The Network which learns to give a definite output by giving a particular Input to the game is known as Policy Network

For Example: Input a1 gives a state s1 (moving up) & Input a2 gives a state s2(going down) in the game.

Also, Some actions increase the points of the player lead to reward r .

Let’s look at some obvious symbols:

Why are we using Discount Factor γ

It is used as a precautionary measure (usually kept below 1). It prevent the reward r to reach infinite.

An infinite reward for a policy will overwhelm our agent & biased towards that specific action, killing the desire to explore unknown areas and actions of the game😵.

But how do we know which state to choose for your next move, eventually leading to the final round?

What is a Value Network?

The value network assigns value/score to the state of the game by calculating an expected cumulative score for the current state s . Every state goes through the value network. The states which gets more reward obviously get more value in the network.

Keep in mind that the reward is expected rewards, because we are choosing the right one from the set of states.

Now, the key objective is always to maximise the reward (aka Markov Decision Process). Actions that results in a good state obviously get greater reward than others.

Since any game is won by following a sequence of actions one after the other. The optimal policy π* of the game consists a number of state-action pairs that helps in winning the game.

The state-action pair that achieve most reward is considered as optimal policy.

The equation for optimal policy is formally written using arg max as:

Therefore, the optimal policy tells us which actions to take to maximises the cumulative discounted reward.

The optimal policy learned by the policy network knows which actions should be performed at the current state to get maximum reward.

If you have any doubt, query or demand, comment down below or tweet me.

Clap it… Share it! Follow me on Medium to get similar fun content.

To get instant notification follow me on Twitter.

Happy to be helpful. kudos.

Previous Stories You will Love:

• Monte Carlo Tree Search
• TensorFlow Image Recognition Python API Tutorial
• Activation Functions in Neural Networks
• DeepMind’s Playing Capture The Flag with Deep Reinforcement Learning

Policy Networks vs Value Networks in Reinforcement Learning was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Games like Tic-Tac-Toe, Rubik’s Cube, Sudoku, Chess, Go and many others have common property that lead to exponential increase in the number of possible actions that can be played. These possible steps increase exponentially as the game goes forward. Ideally if you can predict every possible move and its result that may occur in the future. You can increase your chance of winning.

But since the moves increase exponentially — the computation power that is required to calculate the moves also goes through the roof.

Monte Carlo Tree Search is a method usually used in games to predict the path (moves) that should be taken by the policy to reach the final winning solution.

Before we discover the right path(moves) that will lead us for the win. We first need to arrange the moves of the present state of the game. These moves connected together will look like a tree. Hence the name Tree Search. See the diagram below to understand the sudden exponential increase in moves.

Tree Search Algorithm

It is a method used to search every possible move that may exist after a turn in the game. For Example: There are many different options for a player in the game of tic-tac-toe, which can be visualised using a tree representation. The move can be further increased for the next turn evolving the diagram into a tree.

Brute forcing an exponential increasing tree to find the final solution for a problem by considering every children move/node requires alot of computational power. resulting in extremely slow output.

💡 Faster Tree Search can be achieved by making a policy — giving more importance to some nodes from others & allowing their children nodes to be searched first to reach the correct solution.

But how to find that node which is most favourable to have the correct solution in their children nodes. Let’s find out…

What is Monte Carlo Tree Search ?

MCTS is an algorithm that figures out the best move out of a set of moves by Selecting → Expanding → Simulating → Updating the nodes in tree to find the final solution. This method is repeated until it reaches the solution and learns the policy of the game.

How does Monte Carlo Tree Search Work?

Let’s look at parts of the loop one-by-one.

Selecting👆| This process is used to select a node on the tree that has the highest possibility of winning. For Example — Consider the moves with winning possibility 2/3, 0/1 & 1/2 after the first move 4/6, the node 2/3 has the highest possibility of winning.

The node selected is searched from the current state of the tree and selected node is located at the end of the branch. Since the selected node has the highest possibility of winning — that path is also most likely to reach the solution faster than other path in the tree.

Expanding — After selecting the right node. Expanding is used to increase the options further in the game by expanding the selected node and creating many children nodes. We are using only one children node in this case. These children nodes are the future moves that can be played in the game.

The nodes that are not expanded further for the time being are known are leaves.

Simulating|Exploring 🚀 Since nobody knows which node is the best children/ leaf from the group. The move which will perform best and lead to the correct answer down the tree. But,

How do we find the best children which will lead us to the correct solution?

We use Reinforcement Learning to make random decisions in the game further down from every children node. Then, reward is given to every children node — by calculating how close the output of their random decision was from the final output that we need to win the game.

For example: In the game of Tic-Tac-Toe. Does the random decision to make cross(X) next to previous cross(X) in the game results in three consecutive crosses(X-X-X) that are needed to win the game?

FYI: This can be considered as policy π of RL algorithm. Learn more about Policy & Value Network…

Policy Networks vs Value Networks in Reinforcement Learning

The simulation is done for every children node is followed by their individual rewards.

Let’s say the simulation of the node gives optimistic results for its future and gets a positive score 1/1.

Updating|Back-propagation — Due to the new nodes and their positive or negative scores in the environment. The total scores of their parent nodes must be updated by going back up the tree one-by-one. The new updated scores changes the state of the tree and may also change new future node of the selection process.

After updating all the nodes, the loop again begins by selection the best node in the tree→ expanding of the selected node → using RL for simulating exploration → back-propagating the updated scores → then finally selecting a new node further down the tree that is actual the required final winning result.

For Example: Solved Rubik’s Cube, Sudoku’s correct solution, Killing the King in Chess or

Conclusion

Instead of brute forcing from millions of possible ways to find the right path.

Monte Carlo Tree Search algorithm chooses the best possible move from the current state of Game’s Tree with the help of Reinforcement Learning.

Thanks for reading the article. If you have any doubt or just wants to talk Data Science, write it in the comments below.

Clap it ! Share it! Follow Me!

Follow me on Medium and Twitter to get more content like this.

Previous Stories You will Love:

• Activation Functions in Neural Networks
• Epoch vs Batch Size vs Iterations
• 50 TensorFlow.js API Explained in 5 Minutes | TensorFlow.js Cheetsheet
• TensorFlow on Mobile: Tutorial
• Playing ATARI with 6 Neurons | Open Source Code

Monte Carlo Tree Search was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

DeepMind and other universities has published many End to End Reinforcement Learning papers that are used for problems that can be solved by a single agent. End to End RL algorithms learns both feature representation and decision making in the network by taking pixels as the input and the controls as output.

The real world contain problems that needs multiple individuals acting independently but still collaborating together to achieve a single goal. From playing games like football or basketball to landing a rocket on the moon, a team of individuals works together following a strategy to complete the faster, safer by reducing the risk of failure. This paper can be used to solve many real life tasks, so let’s breakdown the paper to understand their solution.

DeepMind has build an End-to-End population based RL algorithm that tackles the problem successfully using a two-tier optimisation process & by training individuals, acting and learning independent to each other in a team based 3D multi-agent environment (Capture the Flag) working together strategically to achieve a single goal.

This leads to models that suffers from high complexity of the learning problem that arises from the concurrent adaptation of other learning agents in the environment.

The game Capture the Flag has all the traits of the problem above:

1.A 3D first person view multi-agent game.(also can implemented in robotics due to fpv similarity)

2. Agents unaware of each other decisions playing in same environment as opponent or teammate.

3.Strategy based game for learning higher cognitive skills.

Also, the indoor & outdoor theme maps are randomly generated for every game. Two opposing teams consisting of multiple individual players compete to capture each other’s flags by strategically navigating, tagging, and evading opponents. The team with the greatest number of flag captures after five minutes wins the game.

Ad-hoc teams

To develop more generalised policies & learning agent capable of acquiring generalised skills, training fixed teams of agents on a fixed map reduces the diversity in the training data — instead the paper devise an algorithm & training procedure that enables agents to acquire policies that are robust to the variability of maps, number of players, and choice of teammates, a paradigm closely related to ad-hoc team play .

The final win/loss is a delayed episodic signal received from the environment, making it difficult to optimise 1000’s of actions performed by the agent on the basis of only one binary signal at the end of 5 minute game.

This makes it difficult to recognise the actions that were effective in winning the game from the ones that did not help.

We can solve the problem by increasing the number of rewards in the game. By using more frequent Internal rewards, the rewards can be given on the basis of actions performed by the agent.

The memory and long-term temporal reasoning requirements of high level strategic CTF play is met by introducing an agent architecture that features a multi-timescale representation — reminiscent of what has been observed in primate cerebral cortex and an external working memory module — broadly inspired by human episodic memory.

These 3 innovations are integrated with in a scalable, massively distributed & asynchronous computational framework.

In the game, agent receives raw RBG pixels input Xt from the first person perspective at timestep t, produces control action at and receives game points ρt to train agent’s policy π.

The goal of the reinforcement learning is to find a policy that maximises the the expected cumulative γ-discounted reward over a CTF game with T time steps.

π is parameterised by a multi-time scale Recurrent Neural Network with external memory.

The agent’s architecture model constructs a temporally hierarchical temporal representation space and uses recurrent latent variable for sequential agent to promote the use of memory & temporally coherent action sequences.

Probability of Winning

For Ad-hoc teams, the agent’s policy πo should maximise the probability of winning for its teams, { π0, π1, π2, ……… , π(N-1/2) } and its teammates’ policies { π1, π2, ……… , π(N-1/2) }, for a total of N players in the game:

where winning operator > returns 1 if left wins, 0 for losing, and randomly break ties. Also, ω representing specific maps of the games.

For The Win teams — Now that we are using more frequent Internal Rewards rt, we can operationalise the idea of each agent having a denser reward function by specifying rt = w(ρt) based on available game point signals ρt (points are registered for events such as capturing a flag) and allowing the agent to learn the transformation w such that policy optimisation on the internal rewards r t optimises the policy For The Win, giving us the FTW agent.

Traditional methods used for training 1000s of multi-agent environment at such a high scale together is not supported making the methods unstable.

Scalability — Population of total P different agents are trained in parallel with each other by introducing diversity amongst players to stabilise the training(54).

Matchmaking — To improve the skills of the agents, the teammates & opponents are sampled from population P . The agents indexed by ι for a training game using a stochastic matchmaking scheme mp(π) that biases co-players to be of similar skill to player p, increasing the uncertainty.

Agent Skill Level — Agents skill score are estimated online by calculating Elo Score (15) based on the output of training games.

Meta-optimization — It is a method of using one optimizaiton method to train other optimizers. The paper uses population to meta-optimize the internal rewards & hyperparameters of RL process itself. This can be seen as two-tier optimization RL problem. Inner Optimisation aka J inner: The inner optimisation is soled by RL and it maximises J inner, the agents’ expected future discounted internal rewards. Outer Optimisation aka J outer: It is solved with Population Based Training (PBT) and it is maximised w.r.t. internal reward schemes wp and hyperparameters φp , with the inner
optimisation providing the meta transition dynamics.

PBT is an online evolutionary process which adapts internal rewards and hyperparameters and performs model selection by replacing under-performing agents with mutated versions of better agents.

This joint optimization of agents’ policies helps in utilising the potential of combining learning and evolution together which results in maximisation of:

Assessment during Training | Tournament

To assess the generalisation performance of agents during training a tournament is conducted on procedurally generated maps with Ad-hoc matches involving three types of agents.

1. Ablated version of FTW.
2. Quake 3 Arena scripted bots
3. Human participants with with first person game experience.

Results

1. FTW clearly exceeded the win-rate of humans with maps which neither agent nor human had seen previously, i.e. zero-shot generalisation, with a team of two humans on average capturing 16 flags per game less than a team of two FTW agents.

2. Human-agent v/s agent-agent — Only as part of a h-a team did we observe a human winning over an a-a team (5% win probability).

3. Pro gamers v/s FTW — Even after twelve hours of practice the human game testers were only able to win 25% of games against the agent team.

4.Tagging accuracy of agents FTW agents were 80% while humans were lagging at only 48% success. Agents won the match even after their tagging accuracy were artificially reduced to humans accuracy.

Superior observations & controls resolution of humans helped them to surpass FTW agents in Successful long range tagging at 17% & agents at 0.5%.

But FTW again surpassed humans in short range tagging reaction time with 258ms and humans at 559ms.

Firing Neurons in the Network!!! | Knowledge Representation📺

To investigate how network have learned skills with such high -level rich representation.The network was asking past, present or future states of the game. For Example —

• Q: Do I have the Flag? (Present)
• Q: Did I see my teammate recently? (Past)
• Q: Will I be in the opponent’s base soon? (Future)

Similarly, total 200 binary questions was asked based on the features of the games to see the internal representation of the network.

Results

According to the authors, the agent has knowledge of a given feature if logistic regression on the internal state of the agent accurately models the feature. Interestingly, the FTW agent’s representation was found to encode features related to the past particularly well: e.g. the FTW agent was able to classify the state both flags are stray (flags dropped not at base) with 91% AUCROC (area under the receiver operating characteristic curve), compared to 70% with the self-play baseline.

I encourage you to look at the paper to see more detailed stats.

Visualisation

There are many more visualisations in the paper for your help. I have selected few that need less explanation:

Conclusion

In this paper, artificial agent using only pixels and game points as input can learn to play highly competitively in a rich multi-agent environment. This was achieved by combining a number of innovations in agent training- population based training of agents, internal reward optimisation, and temporally hierarchical RL — together with scalable computational architectures.

This paper can be used to solve other problems around you that contain memory and temporally extended interfaces difficulties in their solution. So, I encourage you to read the paper to have fun & understand the methods emerged at the edge of our knowledge about Machine Learning andpush the boundaries by implementing the paper, solving real world problems to live on the edge of the human knowledge.

Thank you for reading the article. To read more breakdown of your favroite papers and get #PaperExplained notifications. Follow me on Medium and Twitter.

Previous Stories you will Love:

• DeepMind’s Amazing Mix & Match RL Technique
• Playing ATARI with 6 Neurons | Open Source Code
• Google X’s Deep Reinforcement Learning in Robotics using Vision
• Activation Functions in Neural Networks
• Epoch vs Batch Size vs Iterations

DeepMind’s Playing Capture The Flag with Deep Reinforcement Learning was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Google is famous for their cutting edge technology and projects including Self Driving Car, Project Loon (Internet balloon), Project Ara and the list goes on. But alot of research goes behind the scenes, which yields in some interesting research papers that literally gives us access and insight on these fun experiments. Encouraging us to replicate the experiments by ourselves and build further to push the boundaries.

The Learning Robots Project by GoogleX has published QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation that tries to master the simple task of picking and grasping different shaped objects. Aiming to replicate some common human activities.

And the success rate is fascinating.

This experiment uses 7 robotic arms that ran 800 hours at a course of 4 months to grasp objects placed infront of them. Each uses a RBG camera (image above) with a resolution of 472x472. The closed-loop vision-based control system is based on a general formulation of robotic manipulation as a Markov Decision Process (MDP).

To be efficient, Off-policy Reinforcement Learning is used which has the ability to learn from data collected hours, days or even weeks ago.

The Qt-Opt algorithm is designed by combining two methods:

1.Large-scale Distributed Optimization (using multiple robots to train model faster, making it a large-scale distributed system)

2.Deep Q-learning algorithm ( RL technique used to learn a policy, which tells an agent which action to take under which circumstances)

What is Qt-Opt?

QT-Opt is a combination of large-scale distributed optimization and Q-learning algorithm resulting in Distributed Q-learning algorithm that supports continuous action spaces, making it well-suited to robotics problems.

To make the robot not go crazy at their initial attempts, the model is initially trained with offline data which doesn’t require real robots and improves the scalability.

For this case, the policy takes an image and returns the sequence on how the arm and gripper should move in 3D space.

RESULTS

The results gives a unbelievable 96% grasp success rate.

The model learned many new things that are sophisticated and borderline humane.

1.When blocks are too close to each other and there is no space for the gripper, policy separates the blocks from the rest before picking it up.

2.Swatting objects from gripper were not the part of dataset but it automatically repositions the gripper for another attempt.

I encourage you to read the research paper for more insight.

Follow me on Medium and Twitter for more #ResearchPaperExplained Notifications.

If you have any Query about the paper or want me to explain your favourite paper, comment below.

Clap it… Share it…. and Clap it again.

Previous Stories you will Love:

• DeepMind’s Amazing Mix & Match RL Technique
• What the Hell is “Tensor” in “TensorFlow”?

Google X’s Deep Reinforcement Learning in Robotics using Vision was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

The doors this paper unlocks for the machine learning community are amazing. The tiny network actually plays Atari game with only 6 neurons comparable and occasionally superior making the previous networks look like jokes. And the best part is that the code is open source on GitHub for everyone to play with. So, let’s get to it…

How is it Possible!!!

We already know that the Deep Reinforcement Learning network that play atari games are basically a dense deep neural network has both the responsibility of internally learning to extract features from the image with the first layers by mapping pixels to intermediate representations (aka feature extraction), allowing the last (few) layer(s) to map these representations to actions and policy or decision making.

Thus both are learned at the same time, making it almost impossible to study the policy separately.

So just by separating the representation learning part from policy learning part relieves the network from constructing intermediate representation allowing it to specialise it in policy approximation which is trained online on the very observations obtained by the interactions.

This paper helps to learn the features simultaneously but separately in complex RL settings.

This allows a smaller policy network to be competitive and solve the problems with much greater sophistication.

We can get better control over the learning just by separating them into two different networks.

To Learn Feature Extraction

The paper uses two new methods based on Vector Quantization called Increasing Dictionary (VQ) and based on Sparse Coding called Direct Residual SC to train the network.

Let’s look at some one liners for quick understanding of the basic concept then I’ll explain the new stuff later in the article.

Vector Quantization is a neural net algorithm used to learn the binary or multi classification by using a list of vectors called dictionary.

Note: I encourage you to learn further about Vector Quantization and Sparse Coding for better understanding.

To Learn Decision Making Policies

policies are trained with a specialized version of Exponential Natural Evolution Strategy capable of coping with dimensionality growth.

Lets Make the System 💻

Now let’s understand how they have implemented this paper. Our system can be coded into 4 simple steps:

1. ENVIRONMENT 👾

The system is build on the OpenAI framework for Atari 2600, with all the limitations of the original console. The observations consists of a [210x180x3] tensor,representing RBG pixels of the screen input.

The output of the network is interpreted (using one-hot encoding) as one of 18 discrete actions, representing the potential inputs from the joystick. The frame-skipping is fixed at 5.

2. COMPRESSOR 💼

You can also call it a pre-processor.

The compressor is a neural network that extracts the low-dimensional code from observations when interfaced with the environment, in an online fashion.

To get the top performance for online learning the paper uses two new methods which are combined together.

Increasing Dictionary Vector Quantization

This version of VQ increases the size of dictionary unlike the vector quantiztion which has fixed dictionary size.

Growth in dictionary size is regulated by a threshold δ, indicating the minimal aggregated residual considered to be a meaningful addition.

Direct Residual Sparse Coding

The performance of the dictionary-base algorithm depends more encoding than on dictionary training. So to improve the performance of IDVQ Direct Residual SC is used. For detailed explanation go here.

3. CONTROLLER 🎮

The controller for all experiments is a single-layer fully-connected recurrent neural network (RNN).

Each neuron receives the following inputs through weighted connections: the inputs to the network, the output of all neurons from the previous activation (initially zeros), and a constant bias (always set to 1).

The number of inputs is equal at any given point in time to the size of the code coming from the varying output of the compressor.

The number of neurons in the output layer is kept equal to the actions for each game, as defined by the ALE simulator. This is as low as 6 in some games, and 18 atmost.

4. OPTIMIZER 🚀

The Optimizer is our learning algorithm, improving the performance of the network over time, in this case an Evolution Strategy called...

Exponential Natural Evolution Strategy 💁Paper

Natural Evolution Strategies is a family of evolutionary algorithms that controls the parameters than the explicit population of individual.

RESULTS

The comparative results over a set of 10 Atari games from the hundreds available (selected by the required criteria) on the OpenAI’s ALE framework are given below.

They compared their work to two recent papers which offer a broad set of results across Atari games.

CONCLUSION

We presented a method to learn effective policies for visual control in reinforcement learning tasks such as Atari games using tiny neural networks that are two orders of magnitude smaller than the deep neural networks typically used for these problems.

This is accomplished by decoupling policy learning from feature construction.

The feature construction is performed outside the network, through a novel and efficient vector quantization algorithm called Increasing Dictionary Vector Quantization, which is trained online (i.e. along the network) on the very observations obtained by the network’s interactions with the environment.

If you like the explanation that Clap and Share this with the community. Follow me on Medium and Twitter to get more #Research Paper Explained Notifications…

If you have any Query about the paper or want me to explain your favourite paper, comment below.

Previous Stories you will Love:

• DeepMind’s Amazing Mix & Match RL Technique
• Epoch vs Batch Size vs Iterations
• 50 TensorFlow.js API Explained in 5 Minutes | TensorFlow.js Cheetsheet
• Activation Functions in Neural Networks
• TensorFlow on Mobile: Tutorial

Playing ATARI with 6 Neurons | Open Source Code was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Mix & Match — Agent Curricula for Reinforcement Learning [ arxiv ]

The Reinforcement Learning Techniques that are used nowadays are very quick and give immediate results for less complex environments using the gradient based policy optimisations.

The Gradient based policies are competitive and not collaborative.

So what if we need long lasting results for more complex environments having agents with complex tasks to perform.

There are many world environment where we do not have the ability to modify the environments and performing Reinforcement Learning on the real world tasks are really time consuming. So to overcome that problem this paper tries to solve that issue with the help for Curriculum Learning and Population Based Training.

Before You Start

Curriculum Learning [Paper]

To design a sequence of task using transfer learning for an agent to train on, such that final performance or learning speed is improved.

Population Based Training for Neural Networks [DeepMind Blog]

PBT has the ability to modify the hyper-parameters of the Network while training.

The Main Idea

The main idea behind the Mix & Match paper is to

Generate multiple variants of RL agents where we don’t have the ability to modify task arranged according to training complexity but only use one MIX & MATCH agent constructed by leveraging different agent varying structurally by their policy generation process.

MIX & MATCH Framework

Multiple agents that are arranged in learning complexity (shown above) are treated as one Mix & Match Agent acting with a mixture of policies.

The knowledge transfer (i.e. distillation) is done in such a way that the complex agents are matched with the simpler ones early on.

Mixing Coefficient is controlled such that ultimately complex target agent is used to generate experience.

Method Details

Let’s assume we are given a sequence of trainable agents 1 ( with corresponding policies π 1 , …, π K , each parametrised with some θ i ⊂ θ which can share some parameters)

The aim is to train π K ,while all remaining agents are there to induce faster/easier learning.

Let’s introduce categorical random variable ‘c’ ∼ Cat(1, …, K|α) Probability mass function p(c = i) = α i ) which will be used to select a
policy at a given time:

The point of Mix & Match is to allow curriculum learning, consequently we need the probability mass function (pmf) of c to be changed over time. Initially the pmf should have α 1 = 1 and near the end of training α K = 1 thus allowing the curriculum of policies from simple π 1 to the target one π K.

Now, it should be trained in such a way which maximises long lasting increase of performance and share knowledge together unlike those gradients.

To address this issue we will be using a Distillation-like Cost (D) which will align the policies together.

The final optimisation problem we consider is just a weighted sum of the original L^RL loss (below), applied to the control policy π^mm and the knowledge transfer loss:

Now, let’s understand the Mix & Match Architecture in Steps →

Policy Mixing

Policy mixing will be done by explicit mixing for the sake of variance reduction.

Knowledge Transfer

For simplicity we consider the case of K = 2. Consider the problem of ensuring that final policy (π2) matches the simpler policy (π1) , while having access to samples from the control policy(πmm) .

For simplicity, we define our M&M loss over the trajectories directly and trajectories (s ∈ S) are sampled from the control policy. The 1 − α term is introduced so that the distillation cost disappears when we switch to π 2 .

Adjusting α (alpha) through training

α is the variable used in the population mass function equation (first equation).

An important component of the proposed method is how to set values of α through time. For simplicity let us again consider the case of K = 2, where one needs just a single α (as c now comes from Bernoulli distribution) which we
treat as a function of time t.

Online hyperparameter tuning → Since α changes through time one cannot use typical hyperparameter tuning techniques as the space of possible values is exponential in number of timesteps (α = (α (1) , · · · , α (T ) ) ∈ 4 Tk−1 ,
where 4 k denotes a k dimensional simplex).

To solve this issue we use Population Based Training (PBT).

Population Based Training and M&M

Population Based Training (PBT) keeps a population of agents, trained in parallel, in order to optimise hyperparameters through time while training and periodically query each other to check how well they are doing relative to others. Badly performing agents copy the weights (neural network parameters) of stronger agents and perform local modifications of their hyperparameters.

This ability of PBT to modify hyperparameters throughout a single training run makes it is possible to discover powerful adaptive strategies e.g. auto-tuned learning rate annealing schedules.

This way poorly performing agents are used to explore the hyperparameters space.

So, we need to define two functions →

eval → which measures how strong a current agent is

explore → which defines how to instigate the hyperparameters.

Note: Keep in mind the PBT agents are the MIX & MATCH Agents which is already a mixture of constituent agents.

Now, we should use one of the two schemes mentioned below, depending on the characteristics of the problem we are interested in.

1. If models is having performance improvement by switching from simple to the more complex model, then

a ) Provide eval with performance (i.e. reward over k episodes) of the mixed policy.

b ) For an explore function for α we randomly add or subtract a fixed value (truncating between 0 and 1). Thus, once there is a significant benefit of switching to more complex one — PBT will do it automatically.

2. Often we want to switch from an unconstrained architecture to some specific, heavily constrained one (where there may not be an obvious benefit in performance from switching).

When training a multitask policy from constituent single-task policies, we can make eval an independent evaluation job which only looks at performance of an agent with α K = 1.

This way we directly optimise for the final performance of the model of interest, but at the cost of additional evaluations needed for PBT.

EXPERIMENTS

Let’s test and analyse whether our new M&M method is working correctly in all circumstances. We will now consider 3 sets of Reinforcement Learning experiments which can be Scaled to large and complex action spaces, Complexities of Agents Architecture and Learning Multitask Policy.

In all the process, we initialise α around 0 and analyse its adaptation through
time.

Note → that even though in the experimental sections we use K = 2, the actual curriculum goes through potentially infinitely many agents being a result of mixing between π 1 and π 2 .

Unless otherwise stated, the eval function returns averaged rewards from last 30 episodes of the control policy.

The Environment Suite of DeepMind Lab offers a range of challenging 3D, first-person view based tasks (see, appendix) for RL agents. Agents perceive 96 × 72 pixel based RGB observations, at 60 fps and can move, rotate, jump and tag built-in bots.

1. Curricula Over Number of Actions Used

The action space is complex represented in 6 dimensional vector where two action group are very high resolution (rotation and looking up/down actions) and remaining four are low resolution actions (moving forward, backward or not moving at all, shooting or not shooting etc).

Here, we use 9 actions to construct π 1, the simple policy (Small action space).

Similarly to the research in continuous control using diagonal Gaussian distributions we use a factorised policy π 2 (a 1 , a 2 , …, a 6 |s) := j=1 π̂ j (a j |s), which we refer to as Big action space.

In order to be able to mix these two policies we map π 1 actions onto the corresponding ones in the action space of π 2.

The mixing of values between networks of agents shown in the figure below.

The paper also discussed Shared Head and Masked KL techniques but both are wore than M&M.

RESULTS

We see that the small action space leads to faster learning but hampers final performance as compared to the big action space.

Mix & Match applied to this setting gets the best of both worlds, it learns fast, and surpasses the final performance of the big action space.

When plotting α through time (Fig. 5 Left) we see that the agent switches fully to the big action space early on, thus showing that small action space was useful only for initial phase of learning. This is further confirmed by looking at how varied the actions taken by the agent are through training.

Fig. 5 (Right) shows how the marginal distribution over actions evolves through time.We see that new actions are unlocked through training, and further that the final distribution is more entropic that the initial one, which means that more entropy leads to more stability.

2. Curricula Over Agent Architecture

We substitute the LSTM with a linear projection from the processed convolutional signal. We share both the convolutional modules as well as the policy/value function projections.

The experiments are focused on various navigation tasks.

On one hand, reactive policies (which can be represented solely by a Feed Forward policy) should learn reasonably quickly to move around and explore, while on the other hand, recurrent networks (which have memory) are needed to maximise the final performance — by either learning to navigate new maze layouts (Explore Object Location Small) or avoiding (seeking) explored unsuccessful (successful) paths through the maze.

RESULTS

M&M applied to the transition of LSTM and FF, leads to significant improvement in the final performance. But not as fast as FF counterpart.

FIG 8 leads to two observations

1. The Green curve switches late to LSTM due to complexity in the level of Game.
2. The model (Blue Curve) has the ability to switch back to the mixture policy if needed.

3. Curricula for Multitask

As a final proof of concept we consider the task of learning a single policy capable of solving multiple RL problems at the same time. The basic approach for this sort of task is to train a model in a mixture of environments or equivalently to train a shared model in multiple environments in parallel.

This type of training can suffer from two drawbacks:

→ It is heavily reward scale dependent, and will be biased towards
high-reward environments.

→ Environments that are easy to train provide a lot of updates for the model and consequently can also bias the solution towards themselves.

Let’s take a look at the results:

One is Explore Object Locations Small, which has high rewards and a steep initial learning curve (lots of reward signal coming from gathering apples).

In remaining two problems training is hard, the agent is interacting with other bots as well as complex mechanics (pick up bonuses, tagging floors, etc.)

The multitask solution focuses on solving the navigation task, while performing comparitively poorly on the more challenging problems.

To apply M&M to this problem we construct one agent per environment
(each acting as π 1 from previous sections) and then one centralised “multitask” agent (π 2 from previous sections). Crucially, agents share convolutional layers but have independent LSTMs.

Training is done in a multitask way, but the control policy in each environment is again a mixture between the task specific π i (the specialist) and π mt (centralised agent).

Since it is no longer beneficial to switch to the centralised policy, we use the
performance of π mt (i.e. the central policy) as the optimisation criterion (eval) for PBT, instead of the control policy.

We evaluate both the performance of the mixture and the centralised agent independently. Fig. 9 shows per task performance of the proposed method. One can notice much more uniform performance —

M&M agent learns to play well in both challenging laser tag environments, while slightly sacrificing performance in a single navigation task.

One of the reasons of this success is the fact that knowledge transfer is done in policy space, which is invariant to reward scaling. While the agent can still focus purely on high reward environments once it has switched to using
only the central policy, this inductive bias in training with M&M ensures a much higher minimum score.

CONCLUSION

Over time the component weightings of this mixture are adapted such that at the end of training we are left with a single active component consisting of the most complex agent. Also,

Improved and Accelerated performance in Complex Environment.

Collection of agents is bound together as a single composite whole using
a mixture policy.

Information can be shared between the components via shared experience or shared architectural elements, and also through a distillation-like KL-matching loss.

I will be posting 2 posts per week so don’t miss the tutorial.

So, follow me on Medium and Twitter to see similar posts.

Any comments or if you have any question, write it in the comment.

Clap it(Multiple Times)! Share it! Follow Me!

Happy to be helpful. kudos…..

Previous stories you will love:

• 50 TensorFlow.js API Explained in 5 Minutes | TensorFlow.js Cheetsheet
• TensorFlow 1.9 has Arrived!
• TensorFlow on Mobile: Tutorial
• Activation Functions in Neural Networks

DeepMind’s Amazing Mix & Match RL Technique was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

At the time of writing this article TensorFlow 1.9 is not updated on their official website but TensorFlow 1.9 rc-0 is out and released on their GitHub.

The Major and Improvements 🚀

1. Update your tf.keras to Keras 2.1 API.
2. Please inherit tf.keras.Model because tfe.Network is deprecated now.
3. Adding support of core feature columns and losses to gradient boosted trees estimators.
4. The Bijectors API supports broadcasting for Bijectors with new API changes.
5. Layered variable are changed in the following conditions

• Using tf.layers in a subclassed tf.keras.Model class. Supported link

Might Break Here

Replace variable_scope('', ...) by variable_scope(tf.get_variable_scope(), ...) if you are opening variable scopes.

Bug Fixes 🐛

tf.data

• The DatasetBase::DebugString() method is now const.
• Added the tf.contrib.data.sample_from_datasets() API for randomly sampling from multiple datasets.

tf.keras

• Move Keras code out of _impl folder and remove API files.
• tf.keras.Model.save_weights now saves in TensorFlow format by default.
• Enable dataset iterators to be passed to tf.keras.Model training/eval methods.

Accelerated Linear Algebra (XLA)

TensorFlow Debugger (tfdbg) CLI

tf.contrib

• Add tf.contrib.data.choose_from_datasets().
• tf.contrib.data.make_csv_dataset() now supports line breaks in quoted strings. Two arguments were removed from make_csv_dataset.
• tf.contrib.framework.zero_initializer supports ResourceVariable.
• Adding “constrained_optimization” to tensorflow/contrib.

Go to the following link to see more Bug Fixes.

Thanks for reading the article, hope you write efficient code with the new features.

For more updates on TensorFlow releases and more follow me on Medium and Twitter.

Clap it! Share it! Follow Me!

Previous stories you will love:

• 50 TensorFlow.js API Explained in 5 Minutes | TensorFlow.js Cheetsheet
• TensorFlow on Mobile: TensorFlow Lite
• TensorFlow on Mobile: Tutorial
• Activation Functions in Neural Networks
• Epoch vs Batch Size vs Iterations

TensorFlow 1.9 has Arrived! was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article I really want to give a look at the TensorFlow.js APIs and understand the library as a whole and understand what are the amazing things it has to offer to the machine learning community.

I know this article should be 5 minutes long but don’t worry it will not take more than 5 minutes to understand the obvious APIs and even though many of them are very obvious from there names, it was important for me to keep it beginner friendly.

I have kept the rest of the article as to-the-point as possible with examples. But If you guys have any doubt, let’s have discussion in the comments.

It will help you to write better & versatile machine learning code for any new future projects.

CREATION 🚀

The APIs help to create things like tensors, clones etc.

1 - tf.tensor ( values , shape? , dtype? )

Creates a Tensor with the specified shape of data type.

// Pass an array of values to create a vector. 
tf.tensor([1, 2, 3, 4]).print();

------RESULT------
Tensor     [1, 2, 3, 4]

// Pass a flat array and specify a shape yourself. 
tf.tensor([1, 2, 3, 4], [2, 2]).print();

------RESULT------
Tensor     [[1, 2],      
            [3, 4]]

Returns → tf.Tensor

Note: They also have shape specified APIs like tf.tensor1d, tf.tensor2d, tf.tensor3d and tf.tensor4d.

2 - tf.buffer ( shape , dtype? , values? )

Creates a Buffer Tensor.

// Create a buffer and set values at particular indices. 
const buffer = tf.buffer([2, 2]); 
buffer.set(3, 0, 0); 
buffer.set(5, 1, 0);  // Convert the buffer back to a tensor. buffer.toTensor().print();

-------RESULT------
Tensor     [[3, 0],
            [5, 0]]

Returns → tf.TensorBuffer

3 - tf.fromPixels ( pixels, numChannels? )

Creates a Tensor from Images.

const image = new ImageData(1, 1); 
image.data[0] = 100; 
image.data[1] = 150; 
image.data[2] = 200; 
image.data[3] = 255;  
tf.fromPixels(image).print();

------RESULT------
Tensor      [ [[100, 150, 200],]]

Returns → tf.Tensor3D

4 - tf.linspace ( start , stop , num )

Creates a Tensor with evenly spaced numbers.

tf.linspace(0, 9, 10).print();

------RESULT------
Tensor     [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Returns → tf.Tensor1D

5 - tf.oneHot ( indices , depth , onValue? , offValue? )

Convert a set of sparse labels to a dense one-hot representation.

tf.oneHot(tf.tensor1d([0, 1], 'int32'), 3).print();

------RESULT------
Tensor     [[1, 0, 0],
            [0, 1, 0]]

Returns → tf.Tensor2D

6 - tf.print ( x , verbose? )

Prints the info. about any Tensor.

const verbose = true; 
tf.tensor2d([1, 2, 3, 4], [2, 2]).print(verbose);

------RESULT------
Tensor   dtype: float32  
         rank: 2  
         shape: [2,2]  
         values:     [[1, 2], 
                      [3, 4]]

Returns → void

7 - tf.variable ( initialValue , trainable? , name? , dtype? )

Creates a Tensor variable with provided initial value.

const x = tf.variable(tf.tensor([1, 2, 3]));
x.assign(tf.tensor([4, 5, 6]));
x.print();

------RESULT------
Tensor     [4, 5, 6]

Returns → tf.Variable

METHODS for tf.Tensor

8 - flatten ( )

Flatten a Tensor to a 1D array

Returns → tf.Tensor1D

9 - asScalar ( )

Converts a size-1 tf.Tensor to a tf.Scalar.

Returns → tf.Scalar

10.1 - as1D ( )

Converts a tf.Tensor to a tf.Tensor1D.

Returns→ tf.Tensor1D

10.2 - as2D (rows , columns)

Converts a tf.Tensor to a tf.Tensor2D.

Returns→ tf.Tensor2D

10.3 - as3D ( rows , columns , depth )

Converts a tf.Tensor to a tf.Tensor3D.

Returns→ tf.Tensor3D

10.4 - as4D ( rows , columns , depth , depth2 )

Converts a tf.Tensor to a tf.Tensor4D.

Returns→ tf.Tensor4D

11 - asType ( dtype )

Casts a tf.Tensor to a specified dtype.

Returns→ this

12 - buffer ( )

Returns a tf.TensorBuffer that holds the underlying data.

Returns→ tf.TensorBuffer

13 - data ( )

The tf.data API helps to build flexible and efficient input pipelines.

Returns→ Promise

14 - dispose ( )

Disposes tf.Tensor from memory.

Returns→ void

15.1 - toFloat ( )

Casts the array to type float32.

Returns→ this

15.2 - toInt ( )

Casts the array to type int32.

Returns→ this

15.3 - toBool ( )

Casts the array to type bool.

Returns→this

16 - reshapeAs ( x )

Reshapes the tensor into the shape of the provided tensor.

Returns→ tf.Tensor

17 - toString ( verbose? )

Returns a human-readable description of the tensor. Useful for logging.

Returns→ string

METHODS for tf.Variable EXTENDS to tf.Tensor

18 - assign (newValue)

Assign a new tf.Tensor to this variable.

Returns→ void

METHODS for tf.TensorBuffer

tf.TensorBuffer allows to set a value before declaring it a tf.Tensor whoch can’t be changed.

19 - set ( value , locs )

Sets a value in the buffer at a given location.

Returns→ void

20 - get ( locs )

Returns the value in the buffer at the provided location.

Returns→ number

21 - toTensor ( )

Creates an immutable tf.Tensor object from the buffer.

Returns→tf.Tensor

TRANSFORAMTIONS

22 - tf.expandDims ( x , axis? )

Returns a tf.Tensor that has expanded rank, by inserting a dimension into the tensor’s shape.

const x = tf.tensor1d([1, 2, 3, 4]); 
const axis = 1; 
x.expandDims(axis).print();

------RESULT------
Tensor     [[1],      [2],      [3],      [4]]

Returns→ tf.Tensor

23 - tf.cast ( x , dtype )

Casts a tf.Tensor to a new dtype.

const x = tf.tensor1d([1.5, 2.5, 3]);
tf.cast(x, 'int32').print();

------RESULT------
Tensor     [1, 2, 3]

Returns→ tf.Tensor

24 - tf.pad ( x , paddings , constantValue? )

Pads a tf.Tensor with a given value and paddings.

const x = tf.tensor1d([1, 2, 3, 4]);
x.pad([[1, 2]]).print();

------RESULT------
Tensor     [0, 1, 2, 3, 4, 0, 0]

Returns→ tf.Tensor

25 - tf.reshape ( x , shape)

Reshapes a tf.Tensor to a given shape.

const x = tf.tensor1d([1, 2, 3, 4]); 
x.reshape([2, 2]).print();

------RESULT------
Tensor     [[1, 2],      
            [3, 4]]

Returns→ tf.Tensor

26 - tf.squeeze ( x , axis? )

Removes dimensions of size 1 from the shape of a tf.Tensor

const x = tf.tensor([1, 2, 3, 4], [1, 1, 4]);
x.squeeze().print();

------RESULT------
Tensor     [1, 2, 3, 4]

Returns→ tf.Tensor

MODELS

Models are a collection of Layers, see Model Creation for details about how Layers can be connected.

27 - tf.model ( ) EXTENDS to CONTAINERS

A model is a data structure that consists of Layers and have nodes defined as inputs and outputs.

Models made with tf.model are just more generic than tf.sequential models.

The following code snippet defines a model consisting of two dense layers (fully connected), with 10 and 4 units.

//Define input, which has a size of 5(not including batch dimension)
const input = tf.input({shape: [5]});  
// First dense layer uses relu activation. 
const denseLayer1 = tf.layers.dense({units: 10, activation: 'relu'}); 
// Second dense layer uses softmax activation. 
const denseLayer2 = tf.layers.dense({units: 2, activation: 'softmax'});  
// Obtain the output symbolic tensor by applying the layers on the input. 
const output = denseLayer2.apply(denseLayer1.apply(input));  
// Create the model based on the inputs. 
const model = tf.model({inputs: input, outputs: output});  
// The model can be used for training, evaluation and prediction. 
// For example, the following line runs prediction with the model on // some fake data. 
model.predict(tf.ones([2, 5])).print();

------RESULT------
Tensor     [[0.3465916, 0.6534085],      [0.3465916, 0.6534085]]

27.1 - compile ( config )

Configures and prepares the model for training and evaluation. Compiling outfits the model with an config (optimizer, loss, and/or metrics). Calling fit or evaluate on an un-compiled model will throw an error.

Returns→ void

27.2 - evaluate ( x , y , config? )

Returns the loss value & metrics values for the model in test mode which are specified during compile(), which needs to happen before calls to evaluate().

const model = tf.sequential({    
layers: [tf.layers.dense({units: 1, inputShape: [10]})] }); model.compile({optimizer: 'sgd', loss: 'meanSquaredError'}); 
const result = model.evaluate(tf.ones([8, 10]), tf.ones([8, 1]),               { batchSize: 4}); 
result.print();
------RESULT------
Tensor     0.00010169133020099252

Returns→ tf.Scalar or tf.Scalar[]

27.3 - predict ( x , config? )

Generates output predictions for the input samples and the computations is done in batches.

const model = tf.sequential({
   layers: [tf.layers.dense({units: 1, inputShape: [10]})]
});
model.predict(tf.ones([8, 10]), {batchSize: 4}).print();
------RESULT------
Tensor    
[[1.8470926],      
[1.8470926],      
[1.8470926],      
[1.8470926],      
[1.8470926],      
[1.8470926],      
[1.8470926],      
[1.8470926]]

Returns→ tf.Tensor or tf.Tensor[]

27.4 - predictOnBatch ( x )

Returns predictions for a single batch of samples where x is a tensor.

const model = tf.sequential({
   layers: [tf.layers.dense({units: 1, inputShape: [10]})]
});
model.predictOnBatch(tf.ones([8, 10])).print();
------RESULT------
Tensor     
[[-1.1825931],      
[-1.1825931],      
[-1.1825931],      
[-1.1825931],      
[-1.1825931],      
[-1.1825931],      
[-1.1825931],      
[-1.1825931]]

Returns→ tf.Tensor or tf.Tensor[ ]

27.5 - fit ( x , y , config? )

Trains the model for a fixed number of epochs (iterations on a dataset).

const model = tf.sequential({      
layers: [tf.layers.dense({units: 1, inputShape: [10]})] }); model.compile({optimizer: 'sgd', loss: 'meanSquaredError'}); 
for (let i = 1; i < 5 ; ++i) {    
const h = await model.fit(tf.ones([8, 10]), tf.ones([8, 1]), 
                          { batchSize: 4, epochs: 3    });    
console.log("Loss after Epoch " + i + " : " + h.history.loss[0]); 
}
------RESULT------
Loss after Epoch 1 : 0.32676371932029724 
Loss after Epoch 2 : 0.016571789979934692 
Loss after Epoch 3 : 0.0008404387044720352 
Loss after Epoch 4 : 0.00004262066795490682

Returns→ Promise

28 - tf.sequential ( ) EXTENDS to tf.Model

A model with a stack of layers, feeding linearly from one to the next.

// Define a model for linear regression.   
const model = tf.sequential();   
model.add(tf.layers.dense({units: 1, inputShape: [1]}));    
//Prepare model for training: Specify the loss and the optimizer.   
model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});    
// Generate some synthetic data for training.   
const xs = tf.tensor2d([1, 2, 3, 4], [4, 1]);   
const ys = tf.tensor2d([1, 3, 5, 7], [4, 1]);    
// Train the model using the data then do inference on a data point //the model hasn't seen:   
await model.fit(xs, ys);   model.predict(tf.tensor2d([5], [1, 1])).print();

------RESULT------
Tensor      [[3.1389987],]

28.1 - add ( layer )

Adds a layer instance on top of the layer stack.

const model = tf.sequential();
model.add(tf.layers.dense({units: 8, inputShape: [1]})); model.add(tf.layers.dense({units: 4, activation: 'relu6'})); model.add(tf.layers.dense({units: 1, activation: 'relu6'}));
  
// Note that the untrained model is random at this point.
 model.predict(tf.randomNormal([10, 1])).print();

------RESULT------
Tensor     [[0.0679427],      
[0.3440365],      
[0.4146437],      
[0        ],      
[0.0855753],      
[0.0688378],      
[0.1540508],      
[0        ],      
[0        ],      
[0        ]]

Returns→ void

28.2 - evaluate ( x , y , config? )

Returns the loss value & metrics values for the model in test mode and Computation is done in batches.

const model = tf.sequential({
   layers: [tf.layers.dense({units: 1, inputShape: [10]})]
});
model.compile({optimizer: 'sgd', loss: 'meanSquaredError'});
const result = model.evaluate(tf.ones([8, 10]), tf.ones([8, 1]), {
   batchSize: 4,
});
result.print();

------RESULT------
Tensor     5.569275379180908

Returns→tf.Scalar or tf.Scalar[]

28.3 - predict ( x , config? )

Generates output predictions for the input samples and Computation is done in batches.

const model = tf.sequential({
   layers: [tf.layers.dense({units: 1, inputShape: [10]})]
});
model.predict(tf.ones([2, 10])).print();

------RESULT------
Tensor     [[-0.6576568],      [-0.6576568]]

Returns→ tf.Tensor or tf.Tensor[]

28.4 - fit ( x , y , config? )

Trains the model for a fixed number of epochs (iterations on a dataset).

const model = tf.sequential({
   layers: [tf.layers.dense({units: 1, inputShape: [10]})]
});
model.compile({optimizer: 'sgd', loss: 'meanSquaredError'});
const history = await model.fit(tf.ones([8, 10]), tf.ones([8, 1]), {
   batchSize: 4,
   epochs: 3
});
console.log(history.history.loss[0]);

------RESULT------
0.486328125

Returns→ Promise

29 - tf.loadModel ( path or IOHandler )

Load a model, including its topology and optionally weights.

Example 1: Save tf.model()’s topology and weights to browser local storage; then load it back.

const model = tf.sequential(
     {layers: [tf.layers.dense({units: 1, inputShape: [3]})]});
console.log('Prediction from original model:');
model.predict(tf.ones([1, 3])).print();

const saveResults = await model.save('localstorage://my-model-1');

const loadedModel = await tf.loadModel('localstorage://my-model-1');
console.log('Prediction from loaded model:');
loadedModel.predict(tf.ones([1, 3])).print();

------RESULT------
Prediction from original model: Tensor      [[0.7820088],] Prediction from loaded model: Tensor      [[0.7820088],]

Example 2: Load a model from user-selected files from HTML file input elements.

// Note: this code snippet will not work without the HTML elements in the page
const jsonUpload = document.getElementById('json-upload');
const weightsUpload = document.getElementById('weights-upload');

const model = await tf.loadModel(
tf.io.browserFiles([jsonUpload.files[0],     weightsUpload.files[0]]));

Example 3: Load a model from an HTTP server.

const model = await
     tf.loadModel('https://storage.googleapis.com/tfjs-models/tfjs/iris_v1/model.json')

MODEL MAGEMENT

30.1 - tf.copyModel ( sourceURL , destURL )

Copy a model from one URL to another.

// First create and save a model. 
const model = tf.sequential(); 
model.add(tf.layers.dense(      {units: 1, inputShape: [10], activation: 'sigmoid'})); 
await model.save('localstorage://demo/management/model1'); 
 
// Then list existing models. 
console.log(await tf.io.listModels());  

// Copy the model, from Local Storage to IndexedDB. 
await tf.io.copyModel('localstorage://demo/management/model1',      'indexeddb://demo/management/model1');  

// List models again. 
console.log(await tf.io.listModels()); 
 
// Remove both models. 
await tf.io.removeModel('localstorage://demo/management/model1'); await tf.io.removeModel('indexeddb://demo/management/model1');

------RESULT------
[object Object] [object Object]

Returns→ Promise

30.2 - tf.listModels ( )

List all models stored in registered storage mediums.

// First create and save a model. 
const model = tf.sequential(); 
model.add(tf.layers.dense(  {units: 1, inputShape: [10], activation:   'sigmoid'})); 
await model.save('localstorage://demo/management/model1');
 
// Then list existing models. 
console.log(await tf.io.listModels());  

// Delete the model. 
await tf.io.removeModel('localstorage://demo/management/model1'); 
 
// List models again. 
console.log(await tf.io.listModels());

------RESULT------
[object Object] [object Object]

Returns:→Promise

30.3 - tf.moveModel ( sourceURL , destURL )

Move a model from one URL to another.

// First create and save a model. 
const model = tf.sequential(); 
model.add(tf.layers.dense({units: 1, inputShape: [10], activation: 'sigmoid'})); 
await model.save('localstorage://demo/management/model1');  

// Then list existing models. 
console.log(await tf.io.listModels()); 

 // Move the model, from Local Storage to IndexedDB. 
await tf.io.moveModel('localstorage://demo/management/model1',      'indexeddb://demo/management/model1');  

// List models again. 
console.log(await tf.io.listModels());  

// Remove the moved model. 
await tf.io.removeModel('indexeddb://demo/management/model1');

------RESULT------
[object Object] [object Object]

Returns→ Promise

30.4 - tf.removeModel ( url )

Remove a model specified by URL from a registered storage medium.

// First create and save a model. 
const model = tf.sequential(); 
model.add(tf.layers.dense(      {units: 1, inputShape: [10], activation: 'sigmoid'})); 
await model.save('localstorage://demo/management/model1');  

// Then list existing models.
 console.log(await tf.io.listModels());  

// Delete the model. 
await tf.io.removeModel('localstorage://demo/management/model1');  

// List models again. 
console.log(await tf.io.listModels());
------RESULT------
[object Object] [object Object]

Returns→ Promise

LAYERS ( Advanced Activation )

31.1 - tf.layers.leakyReLU ( config? )

Input shape: Arbitrary. Use the configuration inputShape when using this layer as the first layer in a model.

(x) = alpha * x for x < 0. f(x) = x for x >= 0.

Output shape→ Same shape as the input.

Returns→ tf.layers.Layer

31.2 - tf.layers.softmax ( config? )

Input shape→ Arbitrary. Use the configuration inputShape when using this layer as the first layer in a model.

Output shape → Same shape as the input.

Returns→ tf.layers.Layer

LAYERS (Basic Functions)

32.1 - tf.layers.dense (units, activation fun., config? )

Creates a dense (fully connected) layer.

output = acticvation(dot(input, kernel(aka weights) ) + bias)

units (number) Positive integer, dimensionality of the output space.

Returns→ tf.layers.Layer

32.2 - tf.layers.dropout (rate , config? )

Applies dropout to the input.

rate (number) Float between 0 and 1. Fraction of the input units to drop.

Returns→ tf.layers.Layer

32.3 - tf.layers.flatten ( config? )

Flattens the input and does not affect the batch size.

It flattens each batch in its inputs to 1D (making the output 2D).

const input = tf.input({shape: [4, 3]});
const flattenLayer = tf.layers.flatten();

// Inspect the inferred output shape of flatten layer, which
// equals `[null, 12]`. The 2nd dimension is 4 * 3, i.e., the result // of the flattening.(The 1st dimension is undermined batch size.)

console.log(JSON.stringify(flattenLayer.apply(input).shape));

------RESULT------
[null,12]

Returns→ tf.layers.Layer

LAYERS (Convolutional)

33 - tf.layers.conv2d ( filters )

This layer creates a convolution kernel that is convolved with the layer input to produce a tensor of outputs.

filters (number) The dimensionality of the output space (i.e. the number of filters in the convolution).

Returns→ tf.layers.Layer

34 - tf.layers.cropping2D ( cropping, config? )

This layer can crop an input at the top, bottom, left and right side of an image tensor.

cropping (number|[number, number]|[[number, number], [number, number]])Dimension of the cropping along the width and the height.

const model = tf.sequential(); model.add(tf.layers.cropping2D({cropping:[[2, 2], [2, 2]],                                 inputShape: [128, 128, 3]})); 

//now output shape is [batch, 124, 124, 3]

Returns→ tf.layers.Layer

LAYERS (Normalization)

35 - tf.layers.batchNormalization ( axis, config? )

Normalize the activations of the previous layer at each batch, i.e. applies a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1.

axis (number) The integer axis that should be normalized (typically the features axis). Defaults to -1.

Returns→ tf.layers.Layer

LAYERS (Pooling)

36 - tf.layers.averagePooling2d ( poolSize, config )

Average pooling operation for spatial data.

poolSize (number or [number, number]) Factors by which to downscale in each dimension [vertical, horizontal]. Expects an integer or an array of 2 integers.

Returns→ tf.layers.Layer

Note: They also have averagePooling1d and globalAveragePooling etc.

37 - tf.layers.maxPooling2d ( poolSize, config )

Max pooling operation for spatial data.

poolSize (number|[number, number]) Factors by which to downscale in each dimension [vertical, horizontal]. Expects an integer or an array of 2 integers.

Returns→ tf.layers.Layer

Note: They also have maxPooling1d and globalMaxPooling etc.

LAYERS (Wrapper)

38 - tf.layers.bidirectional ( layer, config )

It involves duplicating the first recurrent layer in the network so that there are now two layers side-by-side, then providing the input sequence as-is as input to the first layer and providing a reversed copy of the input sequence to the second. This technique helps in LSTMs.

layer (RNN) The instance of an RNN layer to be wrapped.

Returns → Wrapper

39 - tf.layers.timeDistributed ( layer )

The input should be at least 3D, and the dimension of the index 1 will be considered to be the temporal dimension.

Returns→ tf.layers.Layer

LAYERS (Reccurent)

40 - tf.layers.rnn ( cell(tf.RNNCell or tf.RNNCell[ ] ) )

Input shape: 3D tensor with shape .

Returns→ tf.layers.Layer

41 - tf.multiRNNCell (lstmCells, data, c, h)

Computes the next states and outputs of a stack of LSTMCells.

Each cell output is used as input to the next cell

lstmCells Array of LSTMCell functions.

data The input to the cell.

c Array of previous cell states.

h Array of previous cell outputs.

Returns→ [tf.Tensor2D[], tf.Tensor2D[]]

Note: They also have tf.layers.simpleRNNCell ,

tf.layers.stackedRNNCells (config) ,

layers.simpleRNN (config)

42 - tf.layers.lstm (config)

Long-Short Term Memory layer

const lstm = tf.layers.lstm({units: 8, returnSequences: true});  

// Create an input with 10 time steps. 
const input = tf.input({shape: [10, 20]}); 
const output = lstm.apply(input);  

console.log(JSON.stringify(output.shape)); 

// [null, 10, 8]: 1st dimension is unknown batch size; 
// 2nd dimension is the same as the sequence length of [tf.input()]//(#input), due to `returnSequences`: `true`; 
// 3rd dimension is the `LSTMCell`'s number of units.

------RESULT------
[null,10,8]

Returns→ tf.layers.Layer

43 - tf.layers.lstmCell (config)

LSTM Cell is distinct from the RNN subclass LSTM in that its apply method takes the input data of only a single time step and returns the cell's output at the time step, while LSTM takes the input data over a number of time steps.

For example:

const cell = tf.layers.lstmCell({units: 2}); 
const input = tf.input({shape: [10]}); 
const output = cell.apply(input);  console.log(JSON.stringify(output.shape)); 
// [null, 10]: This is cell's output at a single time step. The 1st // dimension is the unknown batch size.

------RESULT------
[null,10]

The most typical use of LSTMCell workflow is to combine a number of cells into a stacked RNN cell. For Example

const cells = [tf.layers.lstmCell({units: 4}),          tf.layers.lstmCell({units: 8}), ]; 
const rnn = tf.layers.rnn({cell: cells, returnSequences: true});  

//Create input with 10 time steps and length-20 vector at each step. 
const input = tf.input({shape: [10, 20]}); 
const output = rnn.apply(input);

console.log(JSON.stringify(output.shape)); 

// [null, 10, 8]: 1st dimension is unknown batch size; 2nd dimension //is the same as the sequence length of [tf.input()](#input), due to //`returnSequences`: `true`; 3rd dimension is the last `lstmCell`'s //number of units.

------RESULT------
[null,10,8]

Returns→ tf.RNNCell

44 - tf.basicLSTMCell ( forgetBias , lstmKernel , lstmBias , data , c , h )

Computes the next state and output of a BasicLSTMCell.

c Previous cell state.

h Previous cell output.

Returns→ [tf.Tensor2D, tf.Tensor2D]

TRAINING (Gradient)

45 - tf.grad ( f )

f stands for mathematical function.

// f(x) = x ^ 2 
const f = x => x.square(); 

// f'(x) = 2x 
const g = tf.grad(f); 
 
const x = tf.tensor1d([2, 3]); 
g(x).print();

------RESULT------
Tensor     [4, 6]

Another Example

// f(x) = x ^ 3 
const f = x => x.pow(tf.scalar(3, 'int32')); 

// f'(x) = 3x ^ 2 
const g = tf.grad(f); 

// f''(x) = 6x 
const gg = tf.grad(g);  
const x = tf.tensor1d([2, 3]); 
gg(x).print();

------RESULT------
Tensor     [12, 18.0000038]

Returns→ (x: tf.Tensor, dy?: tf.Tensor) => tf.Tensor

46 - tf.customGrad ( f )

// Override gradient of our custom x ^ 2 op to be dy * abs(x);
const customOp = tf.customGrad(x => {    
    return {value: x.square(), gradFunc: dy => [dy.mul(x.abs())]}; 
});

const x = tf.tensor1d([-1, -2, 3]); 
const dx = tf.grad(x => customOp(x));  
console.log(`f(x):`); 
customOp(x).print(); 
console.log(`f'(x):`); 
dx(x).print();

------RESULT------
f(x): Tensor     [1, 4, 9] f'(x): Tensor     [1, 2, 3]

Returns→ (…args: tf.Tensor[]) => tf.Tensor

TRAINING (Optimizers)

47 - tf.train.adam ( learningRate?, beta1?, beta2?, epsilon? )

Constructs a AdamOptimizer that uses the Adam algorithm. See https://arxiv.org/abs/1412.6980

Parameters:

learningRate (number) It is used for the Adam gradient descent algorithm. Optional

beta1 (number) The exponential decay rate for the 1st moment estimates. Optional

beta2 (number) The exponential decay rate for the 2nd moment estimates. Optional

epsilon (number) A small constant for numerical stability. Optional

Returns→ AdamOptimizer

48 - tf.train.rmsprop ( learningRate , decay? , momentum? , epsilon? , centered? )

Constructs a tf.RMSPropOptimizer that uses RMSProp gradient descent.

Parameters:

learningRate (number) The learning rate to use for the RMSProp gradient descent algorithm.

decay (number) The discounting factor for the history/coming gradient. Optional

momentum (number) The momentum to use for the RMSProp gradient descent algorithm. Optional

epsilon (number) Small value to avoid zero denominator. Optional

centered (boolean) If true, gradients are normalized by the estimated variance of the gradient. Optional

Returns→ tf.RMSPropOptimizer

TRAINING (Losses)

49 - tf.losses.meanSquaredError (labels, predictions, weights?, reduction?)

Computes the mean squared error between two tensors.

Returns→ tf.Tensor

50 - tf.losses.softmaxCrossEntropy ( labels , logits , dim? )

Computes softmax cross entropy between logits and labels. It measures the probability error in discrete classification tasks in which the classes are mutually exclusive.

For example, each CIFAR-10 image is labeled with one and only one label: an image can be a dog or a truck, but not both.

Returns→ tf.Tensor

Note to Self: Don’t Make Long Lists

— — — — — — — — — BONUS — — — — — — — — — —

PERFORMANCE

tf.tidy ( nameOrFn , fn? , gradMode? )

Executes the provided function f and after it is executed, cleans up all intermediate tensors allocated by f except those returned by f.

Using this method helps avoid memory leaks. In general, wrap calls to operations in tf.tidy() for automatic memory cleanup.

// y = 2 ^ 2 + 1 
const y = tf.tidy(() => {    /
/ a, b, and one will be cleaned up when the tidy ends.    
const one = tf.scalar(1);    
const a = tf.scalar(2);    
const b = a.square();     
console.log('numTensors (in tidy): ' + tf.memory().numTensors);     // The value returned inside the tidy function will return    
// through the tidy, in this case to the variable y.    
return b.add(one); });  
console.log('numTensors (outside tidy): ' + tf.memory().numTensors); y.print();

------RESULT------
numTensors (in tidy): 16 
numTensors (outside tidy): 14 
Tensor     5

Returns→ void|number|string|tf.Tensor|tf.Tensor[]|{[key: string]:tf.Tensor|number|string}

tf.keep ( result )

Keeps a tf.Tensor generated inside a tf.tidy() from being disposed automatically.

let b; 
const y = tf.tidy(() => {    
      const one = tf.scalar(1);    
      const a = tf.scalar(2);     

// b will not be cleaned up by the tidy. a and one will be cleaned //up when the tidy ends.    
b = tf.keep(a.square());     
console.log('numTensors (in tidy): ' + tf.memory().numTensors);     

// The value returned inside the tidy function will return    
// through the tidy, in this case to the variable y.    
return b.add(one); 
});  

console.log('numTensors (outside tidy): ' + tf.memory().numTensors); console.log('y:'); 

y.print(); 
console.log('b:'); 
b.print();

------RESULT------
numTensors (in tidy): 16 
numTensors (outside tidy): 15 
y: Tensor     5 
b: Tensor     4

Returns→ tf.Tensor

tf.memory ( )

Returns memory info at the current time in the program.

Returns→ MemoryInfo

tf.time ( f )

Executes f( ) and returns a promise that resolves with timing information.

const x = tf.randomNormal([20, 20]); 
const time = await tf.time(() => x.matMul(x));  

console.log(`kernelMs: ${time.kernelMs},                       wallTimeMs:   ${time.wallMs}`);

------RESULT------
kernelMs: 0.10000000149011612, wallTimeMs: 33.40000000037253

Returns→ Promise

tf.setBackend ( backendType , safeMode? )

Sets the backend (cpu, webgl, etc) responsible for creating tensors and executing operations on those tensors.

Returns→ void

tf.toPixels ( img , canvas? )

Draws a tf.Tensor of pixel values to a byte array or optionally a canvas.

Also, When the dtype of the input is ‘float32’, we assume values in the range [0–1]. Otherwise, when input is ‘int32’, we assume values in the range [0–255].

canvas (HTMLCanvasElement) The canvas to draw to. Optional

img (tf.Tensor2D or tf.Tensor3D) A rank-2 or rank-3 tensor. If rank-2, draws grayscale. If rank-3, must have depth of 1, 3 or 4. When depth of 1, draws grayscale.

Returns→ Promise

Now… I’m pretty sure that I have missed your favourite API so don’t forget to mention it in the comments or any queries.

Thanks for reading the post.

I hope this article helped you in any way.

Follow me on Medium to get more amazing Tutorials.

Clap it! Share it! Follow Me!

Happy to be helpful. kudos…..

Previous stories you will love:

• TensorFlow on Mobile: TensorFlow Lite
• Epoch vs Batch Size vs Iterations
• TensorFlow on Mobile: Tutorial
• Activation Functions in Neural Networks

50 TensorFlow.js API Explained in 5 Minutes | TensorFlow.js Cheetsheet was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.