Dear readers,

The TensorFlow blog has moved to https://blog.tensorflow.org. Please visit us there.

Thank you to our readers, and to our many guest authors who have created an incredible collection of content representing a diverse body of work. Thanks also to Medium for a great couple of years.

Although we’ve decided to move our blog, articles from the global TensorFlow community — including students, researchers, and companies large and small — will remain an important part of it.

In the future, new articles will be published on https://blog.tensorflow.org. The existing articles here on Medium will remain unchanged.

Over the next month, we plan to gradually republish our collection of articles on Medium to the new blog, so you’ll be able to find everything in one place. We’ve republished a few so far, with many more to go. With their permission, we plan to republish guest articles on the new blog in the same time frame.

Thank you again, and see you at https://blog.tensorflow.org.

Our blog has moved was originally published in TensorFlow on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why does “Yo Google” not work with Google Assistant? After all, it’s only one word different from the phrase “Ok Google”. It’s because Google Assistant is listening for two specific words — or Wake Words. Wake Words are critical to the design of low-power machine learning to process data with a computationally inexpensive model to “wake up” the device for full processing. Audio wake words, such as “Ok Google”, are widely used to wake up AI assistant devices before they process speech using more computationally expensive machine learning models.

With the availability of low-power cameras, a popular application includes using a vision sensor with a microcontroller to classify when an image frame contains a person (or any object of interest). We refer to this application as Visual Wake Words because it enables a device to wake up when a human is present, analogous to how audio wake words are used in speech recognition.

Machine learning at the edge refers to models that run on-device, without connectivity to the cloud. As users deploy models on audio sensors or low-power cameras, microcontrollers provide a good compromise as a low-cost computational platform. However, available machine learning models hardly fit the device constraints (in terms of power consumption and processing power) of microcontrollers as detailed in this paper.

At CVPR 2019, Google organized a new challenge, called the “Visual Wake Words Challenge”, soliciting submissions of tiny vision models for microcontrollers. The challenge was to classify images to two classes (person/not-person) that serves a popular use-case for microcontrollers. Google open-sourced the Visual WakeWords Dataset derived from the COCO dataset: the label 1 corresponded to at least one person (or object of interest) being present in the image, and the label 0 corresponded to the image not containing any objects from person class.

Machine learning for microcontroller devices requires re-thinking the model design, accounting for new tradeoffs between memory footprint, accuracy, and compute costs. Typical microcontrollers have extremely limited on-chip memory (100–320 KB SRAM) and flash storage (256 KB–1 MB). The entire neural network model with its weight parameters and code has to fit within the small memory budget of flash storage. Further, the temporary memory buffer required to store the input and output activations during computation must not exceed the on-chip memory. In Visual Wake Words Challenge, researchers designed models with highest accuracy that fit the microcontroller device constraints: model size is less than 250 KB, peak memory usage is less than 250 KB, inference cost is less than 60 million multiply-adds per inference. The talk from the workshop is available on IEEEtv.

The submissions used optimization techniques such as model pruning and quantization (available from Tensorflow Model Optimization toolkit) and neural architecture search algorithms to design tiny models that fit microcontroller device constraints. To deploy their model on device, users leverage Tensorflow Lite Micro, the ML framework for microcontrollers from TensorFlow team.

The challenge generated a lot of excitement in the research community and received submissions from ARM, Samsung, Qualcomm, MIT, Berkeley, University of Oxford etc. On the Visual WakeWords dataset, top-scoring entries from MIT and Qualcomm achieved a classification accuracy of 94.5% and 95% in two categories: deployable today and deployable in next-generation ML frameworks.

To download the Visual Wake Words dataset and train a model yourself, you can walk through the following tutorial.

Thank you to everyone who worked on this release: Aakanksha Chowdhery, Daniel Situnayake, Pete Warden as well as the following colleagues for their guidance and advice: Jon Shlens, Andrew Howard, Rocky Rhodes, Nat Jeffres, Bo Chen, Mark Sandler, Meghna Natraj, Andrew Selle, Jared Duke.

Visual Wake Words with TensorFlow Lite Micro was originally published in TensorFlow on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hugging Face is the leading NLP startup with more than a thousand companies using their library in production including Bing, Apple, Monzo. All examples used in this tutorial are available on Colab. The links are available in the corresponding sections.

Introduction

Hugging Face is an NLP-focused startup with a large open-source community, in particular around the Transformers library. 🤗/Transformers is a python-based library that exposes an API to use many well-known transformer architectures, such as BERT, RoBERTa, GPT-2 or DistilBERT, that obtain state-of-the-art results on a variety of NLP tasks like text classification, information extraction, question answering, and text generation. Those architectures come pre-trained with several sets of weights. Getting started with Transformers only requires to install the pip package:

pip install transformers

The library has seen super-fast growth in PyTorch and has recently been ported to TensorFlow 2.0, offering an API that now works with Keras’ fit API, TensorFlow Extended, and TPUs 👏. This blog post is dedicated to the use of the Transformers library using TensorFlow: using the Keras API as well as the TensorFlow TPUStrategy to fine-tune a State-of-The-Art Transformer model.

Library & Philosophy

Transformers is based around the concept of pre-trained transformer models. These transformer models come in different shapes, sizes, and architectures and have their own ways of accepting input data: via tokenization.

The library builds on three main classes: a configuration class, a tokenizer class, and a model class.

• The configuration class: the configuration class hosts relevant information concerning the model we will be using, such as the number of layers and the number of attention heads. Below is an example of a BERT configuration file, for the pre-trained weights bert-base-cased. The configuration classes host these attributes with various I/O methods and standardized name properties.

• The tokenizer class: the tokenizer class takes care of converting python string in arrays or tensors of integers which are indices in a model vocabulary. It has many handy features revolving around the tokenization of a string into tokens. This tokenization varies according to the model, therefore each model has its own tokenizer.
• The model class: the model class holds the neural network modeling logic itself. When using a TensorFlow model, it inherits from tf.keras.layers.Layer which means it can be used very simply by the Keras’ fit API or trained using a custom training loop and GradientTape .

Joy in simplicity

The advantage of using Transformers lies in the straight-forward model-agnostic API. Loading a pre-trained model, along with its tokenizer can be done in a few lines of code. Here is an example of loading the BERT and GPT-2 TensorFlow models as well as their tokenizers:

The weights are downloaded from HuggingFace’s S3 bucket and cached locally on your machine. The models are ready to be used for inference or finetuned if need be. Let’s see that in action.

Fine-tuning a Transformer model

Fine-tuning a model is made easy thanks to some methods available in the Transformer library. The next parts are built as such:

• Loading text data and pre-processing it
• Defining the hyper-parameters
• Training (with Keras on CPU/GPU and with TPUStrategy)

Building an input pipeline

We have made an accompanying colab notebook to get you fast on track with all the code. We’ll leverage thetensorflow_datasets package for data loading. Tensorflow-dataset provides us with a tf.data.Dataset , which can be fed into our glue_convert_examples_to_features method.

This method will make use of the tokenizer to tokenize the input and add special tokens at the beginning and the end of sequences (like [SEP], [CLS], </s> or <s> for instance) if such additional tokens are required by the model. This method returns a tf.data.Dataset holding the featurized inputs.

We can then shuffle this dataset and batch it in batches of 32 units using standard tf.data.Dataset methods.

Training with Keras’ fit method

Training a model using Keras’ fit method has never been simpler. Now that we have the input pipeline setup, we can define the hyperparameters, and call the Keras’ fit method with our dataset.

Training with Strategy

Training with a strategy gives you better control over what happens during the training. By switching between strategies, the user can select the distributed fashion in which the model is trained: from multi-GPUs to TPUs.

As of the time of writing, TPUStrategy is the only surefire way to train a model on a TPU using TensorFlow 2. Building a custom loop using a strategy makes even more sense in that regard, as strategies may easily be switched around and training on multi-GPU would require practically no code change.

Building a custom loop requires a bit of work to set-up, therefore the reader is advised to open the following colab notebook to have a better grasp of the subject at hand. It does not go into the detail of tokenization as the first colab has done, but it shows how to build an input pipeline that will be used by the TPUStrategy.

This makes use of Google Cloud Platform bucket as a means to host data, as TPUs are complicated to handle when using local filesystems. The colab notebook is available here.

Transformers now has access to TensorFlow APIs - So what?

The main selling point of the Transformers library is its model agnostic and simple API. Acting as a front-end to models that obtain state-of-the-art results in NLP, switching between models according to the task at hand is extremely easy.

As an example, here’s the complete script to fine-tune BERT on a language classification task(MRPC):

However, in a production environment, memory is scarce. You would like to use a smaller model instead; switching to DistilBERT for example. Simply change the first two lines to these two in order to do so:

As a platform hosting 10+ Transformer architectures, 🤗/Transformers makes it very easy to use, fine-tune and compare the models that have transfigured the deep-learning for NLP field. It serves as a backend for many downstream apps that leverage transformer models and is in use in production by many different companies. We’ll welcome any question or issue you might have on our GitHub repository.

Using TensorFlow 2 for State-of-the-Art Natural Language Processing was originally published in TensorFlow on Medium, where people are continuing the conversation by highlighting and responding to this story.

TensorFlow.js is an open-source library that lets you define, train, and run machine learning models in Javascript. The library has empowered a new set of developers from the extensive JavaScript community to build and deploy machine learning models and enabled new classes of on-device computation. TensorFlow.js runs in all major browsers, server-side in Node.js, and as of recently, in WeChat and React Native providing hybrid-apps access to ML without having to leave the JS ecosystem. Today, we are excited to bring a new way for JavaScript developers to easily train a custom model on your own data and deploy it in your JavaScript application using the GCP AutoML service.

One of the main goals behind TensorFlow.js is to empower JS developers to use the benefits of ML without having to create their own models. There are two existing ways to bring the power of ML to your JavaScript applications: use one of our pre-packaged models, or fine-tune a model on your own data.

For example, our pre-packaged Pose estimation and Body segmentation models allow you to build new forms of user interaction and can form the basis for new accessibility tools. All of these models are published on NPM and you can use them with a few lines of code (no ML knowledge required!):

If you have your own data, you can start with a pre-trained model and fine-tune it. The pre-trained models and the training itself can live in Python or JavaScript. If the training is done in Python, you can seamlessly convert the SavedModel or Keras model to TensorFlow.js using our converter. If the model is already in TensorFlow.js, you don’t need to do any conversion. This codelab shows you how to take MobileNet, an image classification model optimized for edge devices, and fine-tune it on your own image dataset, all in JavaScript.

TensorFlow.js + AutoML

We are excited to announce the integration with Cloud AutoML Vision, a Cloud service that enables developers to train a custom model on their labelled data. If your use-case is related to image classification or object detection, you can use the UI to upload your data, train an edge model and directly export it to a TensorFlow.js model, without any coding. Note that custom models can be directly trained using the open-source TensorFlow.js library (or Python TensorFlow), but AutoML provides an easy-to-use service that automates this process for you.

To make it easy to deploy and run the models efficiently in TensorFlow.js, we published a @tensorflow/tfjs-automl library on NPM. Once you export the model, you can follow our step-by-step guides for image classification and object detection. The guides use the NPM library and show you how to load the pre-trained model in the browser and make a prediction on an image.

If you already have a custom model, AutoML can still be incredibly useful. AutoML Vision Edge models are optimized to a small memory footprint and offer low latency while delivering high accuracy. We worked with several customers and partners and got some amazing early results and feedback. CVP, a business and technology consulting company, has been evaluating some vision scenarios to help with workplace safety on edge-based models used in remote locations. Here is what Cal Zemelman, Director of Data science at CVP said on their results:

“We are working on a smartphone app for a Federal agency that performs image classification to help prevent accidents and improve public safety. The app is mainly written in JavaScript as a Progressive Web App to allow cross-platform usage, so we were training a Keras model and converting it afterwards to TensorFlow.js. With the new “Export to TensorFlow.js” feature, we were able to test out using AutoML to greatly streamline our model creation workflow. Previously, our best model was a retrained ResNet50 which got to about 91% accuracy. The middle of the road balanced performance model from AutoML Image Classification hit ~99% accuracy in 5 node hours. It is also 1/5 the size of the previous model and we saw a 60% improvement in the speed of inference.”

We are very excited about the early results our customers and partners are seeing and look forward to hearing more with this release.

You can train an AutoML Vision Edge model today. Once you train the model, you can export the TensorFlow.js model to your client-side application. With AutoML Vision Edge, you only pay for training for any usage beyond 15 node hours you get for free. There are no charges for exporting or client side predictions.

Next steps

• To learn more about AutoML, click here.
• For object detection, follow the step-by-step guides on training and deploying to TensorFlow.js
• For image classification, follow the step-by-step guides on training and deploying to TensorFlow.js
• Learn more about the AutoML NPM library via the official documentation.

Build and deploy TensorFlow.js models with the power of AutoML was originally published in TensorFlow on Medium, where people are continuing the conversation by highlighting and responding to this story.

Whether you’re en route to TensorFlow World or you’re unable to make it, learn more below about our new demo experience and explore new GitHub code repos. Be sure to follow #MissionToTensorFlowWorld on Twitter to see this experience in action!

Celebrating the 50th anniversary of the Apollo 11 mission, the first manned mission to land on the Moon, Mission to TensorFlow World is TensorFlow’s latest space game experience that leverages the entire product ecosystem: TensorFlow Core, TensorFlow.js, TensorFlow Lite, and TensorFlow Extended (TFX) to control a spacecraft traveling through space. Each station uses different versions of TensorFlow on different platforms to power mini demos that influence a central visualization.

Pose Shield (TensorFlow.js)

On its Mission to TensorFlow World, the spacecraft must pass through a dangerous asteroid field. Using PoseNet and TensorFlow.js, you can enable and disable shield emitters around the periphery of a spacecraft using your arms attempting to block incoming asteroids. Are you quick enough and accurate enough to save the ship from damage? Play the game at g.co/PoseShield.

Emojicules (TensorFlow Lite)

Mechanical failures are inevitable on any space journey, and, unfortunately, the spacecraft has experienced a loss of cabin pressure. Using Emotion Classification and TensorFlow Lite, you must match your facial expression to emoji faces — representing the molecules that make up breathable air (Nitrogen, Oxygen, Carbon Dioxide, and Argon) — to capture those molecules and repressurize the cabin. Repo coming soon!

Startographer (TensorFlow Core, TFX)

As the spacecraft enters new star systems on its Mission to TensorFlow World, charting the most direct course is challenging. Using XLNet and TensorFlow Core, planets are assigned conceptual phrases that must be matched with a synonym in order to pass. Startographer begins with a short animation describing the purpose and usage of TFX as a critical validation tool for any TensorFlow implementation. Within the gameplay itself, a TFX calibration animation displays a new planet group referencing TFX’s ability to identify and discard bad data. How many star systems can you travel through before time runs out? Repo coming soon!

For those who will be at TensorFlow world, we will see you soon! For those who can’t make it, be sure to watch the livestream on Wednesday (10/30).

“Mission to TensorFlow World” Demo was originally published in TensorFlow on Medium, where people are continuing the conversation by highlighting and responding to this story.

Arduino is on a mission to make Machine Learning simple enough for anyone to use. We’ve been working with the TensorFlow Lite team over the past few months and are excited to show you what we’ve been up to together: bringing TensorFlow Lite Micro to the Arduino Nano 33 BLE Sense. In this article, we’ll show you how to install and run several new TensorFlow Lite Micro examples that are now available in the Arduino Library Manager.

The first tutorial below shows you how to install a neural network on your Arduino board to recognize simple voice commands.

Next, we’ll introduce a more in-depth tutorial you can use to train your own custom gesture recognition model for Arduino using TensorFlow in Colab. This material is based on a practical workshop held by Sandeep Mistry and Don Coleman, an updated version of which is now online.

If you have previous experience with Arduino, you may be able to get these tutorials working within a couple of hours. If you’re entirely new to microcontrollers, it may take a bit longer.

We’re excited to share some of the first examples and tutorials, and to see what you will build from here. Let’s get started!

Note: the following projects are based on TensorFlow Lite for Microcontrollers which is currently experimental within the TensorFlow repo. This is still a new and emerging field!

Microcontrollers and TinyML

Microcontrollers, such as those used on Arduino boards, are low-cost, single chip, self-contained computer systems. They’re the invisible computers embedded inside billions of everyday gadgets like wearables, drones, 3D printers, toys, rice cookers, smart plugs, e-scooters, washing machines. The trend to connect these devices is part of what is referred to as the Internet of Things.

Arduino is an open-source platform and community focused on making microcontroller application development accessible to everyone. The board we’re using here has an Arm Cortex-M4 microcontroller running at 64 MHz with 1MB Flash memory and 256 KB of RAM. This is tiny in comparison to Cloud, PC, or Mobile but reasonable by microcontroller standards.

There are practical reasons you might want to squeeze ML on microcontrollers, including:

• Function — wanting a smart device to act quickly and locally (independent of the Internet).
• Cost — accomplishing this with simple, lower cost hardware.
• Privacy — not wanting to share all sensor data externally.
• Efficiency — smaller device form-factor, energy-harvesting or longer battery life.

There’s a final goal which we’re building towards that is very important:

• Machine learning can make microcontrollers accessible to developers who don’t have a background in embedded development

On the machine learning side, there are techniques you can use to fit neural network models into memory constrained devices like microcontrollers. One of the key steps is the quantization of the weights from floating point to 8-bit integers. This also has the effect of making inference quicker to calculate and more applicable to lower clock-rate devices.

TinyML is an emerging field and there is still work to do — but what’s exciting is there’s a vast unexplored application space out there. Billions of microcontrollers combined with all sorts of sensors in all sorts of places which can lead to some seriously creative and valuable Tiny ML applications in the future.

What you need to get started

• An Arduino Nano 33 BLE Sense board
• A Micro USB cable to connect the Arduino board to your desktop machine
• To program your board, you can use the Arduino Web Editor or install the Arduino IDE. We’ll give you more details on how to set these up in the following section

The Arduino Nano 33 BLE Sense has a variety of onboard sensors meaning potential for some cool Tiny ML applications:

• Voice — digital microphone
• Motion — 9-axis IMU (accelerometer, gyroscope, magnetometer)
• Environmental — temperature, humidity and pressure
• Light — brightness, color and object proximity

Unlike classic Arduino Uno, the board combines a microcontroller with onboard sensors which means you can address many use cases without additional hardware or wiring. The board is also small enough to be used in end applications like wearables. As the name suggests, it has Bluetooth LE connectivity so you can send data (or inference results) to a laptop, mobile app or other BLE boards and peripherals.

Tip: Sensors on a USB stick

Connecting the BLE Sense board over USB is an easy way to capture data and add multiple sensors to single board computers without the need for additional wiring or hardware — a nice addition to a Raspberry Pi, for example.

TensorFlow Lite for Microcontrollers examples

The inference examples for TensorFlow Lite for Microcontrollers are now packaged and available through the Arduino Library manager making it possible to include and run them on Arduino in a few clicks. In this section we’ll show you how to run them. The examples are:

• micro_speech — speech recognition using the onboard microphone
• magic_wand — gesture recognition using the onboard IMU
• person_detection — person detection using an external ArduCam camera

For more background on the examples you can take a look at the source in the TensorFlow repository. The models in these examples were previously trained. The tutorials below show you how to deploy and run them on an Arduino. In the next section, we’ll discuss training.

How to run the examples using Arduino Create web editor

Once you connect your Arduino Nano 33 BLE Sense to your desktop machine with a USB cable you will be able to compile and run the following TensorFlow examples on the board by using the Arduino Create web editor:

Focus on the speech recognition example: micro_speech

One of the first steps with an Arduino board is getting the LED to flash. Here, we’ll do it with a twist by using TensorFlow Lite Micro to recognise voice keywords. It has a simple vocabulary of “yes” and “no”. Remember this model is running locally on a microcontroller with only 256KB of RAM, so don’t expect commercial ‘voice assistant’ level accuracy — it has no Internet connection and on the order of 2000x less local RAM available.

Note the board can be battery powered as well. As the Arduino can be connected to motors, actuators and more, this offers the potential for voice controlled projects.

How to run the examples using the Arduino IDE

Alternatively, you can use try the same inference examples using Arduino IDE application.

First, follow the instructions in the next section Setting up the Arduino IDE.

In the Arduino IDE, you will see the examples available via the File > Examples > Arduino_TensorFlowLite menu in the ArduinoIDE.

Select an example and the sketch will open. To compile, upload, and run the examples on the board, and click the arrow icon:

For advanced users who prefer a command line, there is also the arduino-cli.

Training a TensorFlow Lite Micro model for Arduino

Next, we will use ML to enable the Arduino board to recognise gestures. We’ll capture motion data from the Arduino Nano 33 BLE Sense board, import it into TensorFlow to train a model, and deploy the resulting classifier onto the board.

The idea for this tutorial was based on Charlie Gerard’s awesome Play Street Fighter with body movements using Arduino and Tensorflow.js. In Charlie’s example, the board is streaming all sensor data from the Arduino to another machine which performs the gesture classification in Tensorflow.js. We take this further and “TinyML-ify” it by performing gesture classification on the Arduino board itself. This is made easier in our case as the Arduino Nano 33 BLE Sense board we’re using has a more powerful Arm Cortex-M4 processor, and an on-board IMU.

We’ve adapted the tutorial below, so no additional hardware is needed — the sampling starts on detecting movement of the board. The original version of the tutorial adds a breadboard and a hardware button to press to trigger sampling. If you want to get into a little hardware, you can follow that version instead.

Setting up the Arduino IDE

Following the steps below sets up the Arduino IDE application used to both upload inference models to your board and download training data from it in the next section. There are a few more steps involved than using Arduino Create web editor because we will need to download and install the specific board and libraries in the Arduino IDE.

• Download and install the Arduino IDE from https://arduino.cc/downloads
• Open the Arduino application you just installed
• In the Arduino IDE menu select Tools > Board > Boards Manager…
• Search for “Nano BLE” and press install on the board
• It will take several minutes to install
• When it’s done close the Boards Manager window

• Now go to the Library Manager Tools > Manage Libraries…
• Search for and install the Arduino_TensorFlowLite library
• Next search for and install the Arduino_LSM9DS1 library:

• Finally, plug the micro USB cable into the board and your computer
• Choose the board Tools > Board > Arduino Nano 33 BLE
• Choose the port Tools > Port > COM5 (Arduino Nano 33 BLE)
• Note that the actual port name may be different on your computer

There are more detailed Getting Started and Troubleshooting guides on the Arduino site if you need help.

Streaming sensor data from the Arduino board

First, we need to capture some training data. You can capture sensor data logs from the Arduino board over the same USB cable you use to program the board with your laptop or PC.

Arduino boards run small applications (also called sketches) which are compiled from .ino format Arduino source code, and programmed onto the board using the Arduino IDE or Arduino Create.

We’ll be using a pre-made sketch IMU_Capture.ino which does the following:

• Monitor the board’s accelerometer and gyroscope
• Trigger a sample window on detecting significant linear acceleration of the board
• Sample for one second at 119Hz, outputting CSV format data over USB
• Loop back and monitor for the next gesture

The sensors we choose to read from the board, the sample rate, the trigger threshold, and whether we stream data output as CSV, JSON, binary or some other format are all customizable in the sketch running on the Arduino. There is also scope to perform signal preprocessing and filtering on the device before the data is output to the log — this we can cover in another blog. For now, you can just upload the sketch and get to sampling.

To program the board with this sketch in the Arduino IDE:

• Download IMU_Capture.ino and open it in the Arduino IDE
• Compile and upload it to the board with Sketch > Upload

Visualizing live sensor data log from the Arduino board

With that done, we can now visualize the data coming off the board. We’re not capturing data yet — this is just to give you a feel for how the sensor data capture is triggered and how long a sample window is. This will help when it comes to collecting training samples.

• In the Arduino IDE, open the Serial Plotter Tools > Serial Plotter
• If you get an error that the board is not available, reselect the port:
• Tools > Port > portname (Arduino Nano 33 BLE)
• Pick up the board and practice your punch and flex gestures
• You’ll see it only sample for a one second window, then wait for the next gesture
• You should see a live graph of the sensor data capture (see GIF below)

When you’re done be sure to close the Serial Plotter window — this is important as the next step won’t work otherwise.

Capturing gesture training data

To capture data as a CSV log to upload to TensorFlow, you can use Arduino IDE > Tools > Serial Monitor to view the data and export it to your desktop machine:

• Reset the board by pressing the small white button on the top
• Pick up the board in one hand (picking it up later will trigger sampling)
• In the Arduino IDE, open the Serial Monitor Tools > Serial Monitor
• If you get an error that the board is not available, reselect the port:
• Tools > Port > portname (Arduino Nano 33 BLE)
• Make a punch gesture with the board in your hand (Be careful whilst doing this!)
• Make the outward punch quickly enough to trigger the capture
• Return to a neutral position slowly so as not to trigger the capture again
• Repeat the gesture capture step 10 or more times to gather more data
• Copy and paste the data from the Serial Console to new text file called punch.csv
• Clear the console window output and repeat all the steps above, this time with a flex gesture in a file called flex.csv
• Make the inward flex fast enough to trigger capture returning slowly each time

Note the first line of your two csv files should contain the fields aX,aY,aZ,gX,gY,gZ

Linux tip: if you prefer you can redirect the sensor log output from the Arduino straight to a .csv file on the command line. With the Serial Plotter / Serial Monitor windows closed use:

$ cat /dev/cu.usbmodem[nnnnn] > sensorlog.csv

Training in TensorFlow

We’re going to use Google Colab to train our machine learning model using the data we collected from the Arduino board in the previous section. Colab provides a Jupyter notebook that allows us to run our TensorFlow training in a web browser.

The Colab will step you through the following:

• Setup Python Environment
• Upload the punch.csv and flex.csv data
• Parse and prepare the data
• Build & Train the Model
• Convert the Trained Model to TensorFlow Lite
• Encode the Model in an Arduino Header File

The final step of the colab is generates the model.h file to download and include in our Arduino IDE gesture classifier project in the next section:

Let’s open the notebook in Colab and run through the steps in the cells — arduino_tinyml_workshop.ipynb

Classifying IMU Data

Next we will use model.h file we just trained and downloaded from Colab in the previous section in our Arduino IDE project:

1. Open IMU_Classifier.ino in the Arduino IDE.
2. Create a new tab in the IDE. When asked name it model.h

3. Open the model.h tab and paste in the version you downloaded from Colab

4. Upload the sketch: Sketch > Upload

5. Open the Serial Monitor: Tools > Serial Monitor

6. Perform some gestures

7. The confidence of each gesture will be printed to the Serial Monitor (0 = low confidence, 1 = high confidence)

Congratulations, you’ve just trained your first ML application for Arduino!

For added fun, the Emoji_Button.ino example shows how to create a USB keyboard that prints an emoji character in Linux and macOS. Try combining the Emoji_Button.ino example with the IMU_Classifier.ino sketch to create a gesture controlled emoji keyboard 👊.

Conclusion

It’s an exciting time with a lot to learn and explore in Tiny ML. We hope this blog has given you some idea of the potential and a starting point to start applying it in your own projects. Be sure to let us know what you build and share it with the Arduino community.

For a comprehensive background on TinyML and the example applications in this article, we recommend Pete Warden and Daniel Situnayake’s new O’Reilly book “TinyML: Machine Learning with TensorFlow on Arduino and Ultra-Low Power Microcontrollers”

How-to Get Started with Machine Learning on Arduino was originally published in TensorFlow on Medium, where people are continuing the conversation by highlighting and responding to this story.

TensorFlow 2.0 is driven by the community telling us they want an easy-to-use platform that is both flexible and powerful, and which supports deployment to any platform. TensorFlow 2.0 provides a comprehensive ecosystem of tools for developers, enterprises, and researchers who want to push the state-of-the-art in machine learning and build scalable ML-powered applications.

Coding with TensorFlow 2.0

TensorFlow 2.0 makes development of ML applications much easier. With tight integration of Keras into TensorFlow, eager execution by default, and Pythonic function execution, TensorFlow 2.0 makes the experience of developing applications as familiar as possible for Python developers. For researchers pushing the boundaries of ML, we have invested heavily in TensorFlow’s low-level API: We now export all ops that are used internally, and we provide inheritable interfaces for crucial concepts such as variables and checkpoints. This allows you to build onto the internals of TensorFlow without having to rebuild TensorFlow.

To be able to run models on a variety of runtimes, including the cloud, web, browser, Node.js, mobile and embedded systems, we have standardized on the SavedModel file format. This allows you to run your models with TensorFlow, deploy them with TensorFlow Serving, use them on mobile and embedded systems with TensorFlow Lite, and train and run in the browser or Node.js with TensorFlow.js.

For high-performance training scenarios, you can use the Distribution Strategy API to distribute training with minimal code changes and attain great out-of-the-box performance. It supports distributed training with Keras’ Model.fit and also supports custom training loops. Multi-GPU support is now available, and you can learn more about using GPUs on Google Cloud here. Cloud TPU support is coming in a future release. Check out the distributed training guide for more details.

TensorFlow 2.0 offers many performance improvements on GPUs. TensorFlow 2.0 delivers up to 3x faster training performance using mixed precision on Volta and Turing GPUs with a few lines of code, used for example in ResNet-50 and BERT. TensorFlow 2.0 is tightly integrated with TensorRT and uses an improved API to deliver better usability and high performance during inference on NVIDIA T4 Cloud GPUs on Google Cloud.

“Machine learning on NVIDIA GPUs and systems allows developers to solve problems that seemed impossible just a few years ago,” said Kari Briski, Senior Director of Accelerated Computing Software Product Management at NVIDIA. “TensorFlow 2.0 is packed with many great GPU acceleration features, and we can’t wait to see the amazing AI applications the community will create with these updated tools.”

Effective access for training and validation data is paramount to being effective when building models in TensorFlow. We introduced TensorFlow Datasets, giving a standard interface to a plethora of datasets containing a variety of data types such as images, text, video, and more.

While the traditional Session-based programming model is still maintained, we recommend using regular Python development with eager execution. The tf.function decorator can be used to convert your code into graphs which can be executed remotely, serialized, and optimized for performance. This is complemented by Autograph, which can convert regular Python control flow directly into TensorFlow control flow.

And of course, if you have used TensorFlow 1.x and are looking for a migration guide to 2.0, we have published one here. The TensorFlow 2.0 release also includes an automatic conversion script to help get you started.

We have worked with many users inside Google and from the TensorFlow community to test TensorFlow 2.0 features and have been thrilled with the feedback. As an example, the Google News team launched a BERT-based language understanding model in TensorFlow 2.0 that significantly improved story coverage. TensorFlow 2.0 provides easy-to-use APIs with the flexibility to implement new ideas quickly. Model training and serving was seamlessly integrated into the existing infrastructure.

Also, ML isn’t just for Python developers — using TensorFlow.js, training and inference is available to JavaScript developers, and we continue to invest in Swift as a language for building models with the Swift for TensorFlow library.

There’s a lot to unpack here, so to assist, we’ve created a handy guide on how to be effective with all that’s new in TensorFlow 2.0. To make it even easier to get started with TensorFlow 2.0, we are releasing reference implementations of several commonly used ML models using the 2.0 API here.

Additionally, to learn how to build applications using TensorFlow 2.0, check out the online courses we have created together with deeplearning.ai and Udacity.

To get started quickly try Google Cloud’s Deep Learning VM Images — no setup required, pre-configured virtual machines to help you build your TensorFlow 2.0 deep learning projects.

Learn More

Head over to tensorflow.org for more about TensorFlow 2.0, including how to download and get started with coding ML applications. Finally, for more exciting announcements, talks, and hands-on training and presentations about TensorFlow 2.0 and its ecosystem, please join us in person at TensorFlow World on Oct 28–31 in Santa Clara, CA. We hope to see you there!

TensorFlow 2.0 is now available! was originally published in TensorFlow on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Mercado Libre is the leading marketplace platform in Latin America, reaching millions of users selling and buying tens of millions of different items a day. From a shipping perspective, one of the most important pieces of information for an item is dimensions and weight since they are used for predicting costs and forecasting the occupancy of the fulfillment centers. As a user-driven marketplace, this information is not always available, and we have found that we can optimize our pipeline by predicting it in advance.

Mercado Libre is a general marketplace, so an item could be either a new cellphone (between 7 - 14 ounces, and approximately 5″ x 5″ x 2″), or a used dishwasher weighing 175 pounds and measuring 24″ x 24″ x 35″.

Initially, we implemented a lookup mechanism, which consisted of determining measure and weight based on previous measurements of the item in our fulfillment, cross-docking, and validating third party carriers, and after an outlier filtering process. Of course, this process had low coverage (the percentage of items already measured was low).

As a second measure, we built simple models based on statistical summaries of single product features, like category and brand/model, and used product similarity metrics to map new items to existing ones. These methods had poor accuracy in general, and there were a large number of cases where no prediction was possible because there weren’t adequate samples.

The need arose for a regression model, able to infer measurements from information like the item title, description, category, and a few other attributes. Importantly, these attributes include both structured and unstructured data, which are difficult to model with traditional statistical methods. The model predicts and saves 3k predictions a minute (4M a day), and the API serves (mostly pre-saved) 200k requests a minute (288M a day).

Data and features

When preparing the next iteration, after reviewing the listing publication process, we had the certainty that at least the listing title, item description (free text), along with the category and the product brand were likely to be key features for the model. Of course, the data had to be cleaned up and prepared to be useful for our project. The software stack for the ETL part of the project, was based on scipy, pandas, numpy and scikit-learn. Once we had decided on features, it was time to vectorize the data.

To encode the two main text based features, we tokenized and lower cased all the words. Next, we needed to find a suitable vector representation. For this first iteration of this project, we decided to experiment with a method using keywords and TF-idf, rather than an LSTM which we will explore in the future. TF-idf is a vectorization technique, applied to text data, which converts tokenized documents into a matrix.

The Model

The model we used was a Multi-Layer Perceptron, implemented in keras with the Tensorflow backend. Hyperparameters and architecture details were chosen by performing a grid search on learning rate, dropout, number of layers, number of kernels and kernel dimensions (we now know that random search is a much better approach).

Regarding the loss function, we tried several experiments to see which one was more appropriate:

The mean_squared_error provided a measure that, even when heavily penalizing the errors, overshot to fit outliers. The mean_absolute_percentage_error preserved especially the smaller items zone from outliers (with the lower 1Kg covering 50% of the items). The mean_absolute_error was useful for general and broad optimization, across all item classes.

Serving predictions

The model is currently being used in the product submission page and the fulfillment center packaging calculator, to forecast the occupancy of the shelves and the right envelope to use at the final product packaging stage.

The API currently serves 3k predictions a minute or 4M a day. The inference time for our model is 30ms and we serve it on a 64GB RAM + 6 cores instance with 13 ~900MB workers.

Here is an example Json response:

Moving forward, we plan to run additional experiments to optimize model quality and to refine the results based on feedback from the fulfillment center. Working with TensorFlow enabled us to rapidly design, iterate, and deploy models to production.

Acknowledgments:

• The whole Pymes Team, especially Mirko Panozzo, Conrado García Berrotarán, Kevin Clemoveki, Diego Piloni, Martín Ciruzzi and Esteban Tundidor.
• Francisco Ingham for helping with the grammar, style and contents.
• Joaquin Verdinelli for coordinating the redaction and communication with Google.
• Constanza Stahl for helping with the graphics.
• Cynthia Bustamante for general coordination help

Using TensorFlow to predict product weight and dimensions was originally published in TensorFlow on Medium, where people are continuing the conversation by highlighting and responding to this story.

TensorFlow Extended (TFX) is a platform for creating production-ready ML pipelines. TFX was created by Google and provides the backbone of Google’s ML services and applications, and we’ve been open sourcing TFX for everyone who needs to create production ML pipelines.

TFX can be extended and customized in several ways, including developing new components and including them in your pipeline. A TFX pipeline is a series of TFX components, each of which performs a separate task, which are sequenced as a directed acyclic graph (DAG). In this post we’ll present an example to illustrate the process of developing a new TFX component. Watch for more posts that will discuss additional ways to extend and customize TFX.

Scenario

We want to replace the behavior of the existing TFX Trainer component with a new component that includes an executor which submits a job to run the same training on the Google Cloud Platform (GCP). Since the upstream and downstream semantics will remain unchanged, we will reuse the existing Trainer component and replace the behavior of the executor.

Anatomy of a Component

TFX components consist of three main pieces:

• Driver
• Executor
• Publisher

Driver and Publisher

The driver supplies metadata to the executor by querying the ML Metadata (MLMD) store and the publisher takes the results of the executor and updates the metadata store. As a developer, you will typically not need to interact with the driver and publisher directly, but messages logged by the driver and publisher may be useful during debugging.

Executor

The executor is where a component performs its processing. As a developer you write code which runs in the executor, based on the requirements of the classes which implement the type of component that you’re working with. For example, when you’re working on a Transform component you will need to develop a preprocessing_fn. Executors consume and create artifacts, which are kept in the metadata store.

Adding a custom executor

Creating a custom executor

To create the custom executor, we start with a copy of the current Trainer executor and then make the modifications to our custom executor to initiate a training job on Google Cloud AI Platform. Much of the basic executor structure will remain the same as the inputs, outputs, and execution parameters will be the same. The changes will be in how the inputs are processed and the outputs are generated. This is achieved by creating a new Executor class that extends tfx.components.base.base_executor.BaseExecutor and implements Do().

class Executor(base_executor.BaseExecutor):
  """Start a trainer job on Google Cloud AI Platform."""

  def Do(self, input_dict,
         output_dict,
         exec_properties):
    """Starts a trainer job on Google Cloud AI Platform.

Don’t forget to test it before moving on to the next step! We have created a convenience script for you to try out your executor before putting it into production. You should write similar code to exercise unit tests for your code. As with any production software deployment, when developing for TFX you should make sure to have good test coverage and a strong CI/CD framework.

Override the executor used by the Trainer component

In order to do this, we will replace the default trainer executor used by TFX with the new custom executor which will create the training job on Google Cloud AI Platform. This is achieved with the optional executor_class component parameter.

from tfx.extensions.google_cloud_ai_platform.trainer
import executor as ai_platform_trainer_executor
...
trainer = Trainer(
    ...,
    executor_class=ai_platform_trainer_executor.Executor,
}

That’s it! Now when the Trainer component is called by the workflow engine, it will run the custom executor instead of the default executor, while creating and consuming the same ML Metadata artifacts as the default executor.

Pass the component arguments to your trainer

TFX executors are self-contained binaries focused on running a single step of the ML pipeline. Custom executors require the same three parameters as all other TFX executors: input_dict, output_dict, exec_properties. More details on the semantics of these parameters can be found in the BaseExecutor class.

When processing data flowing through your TFX pipeline you will often typically want to read input data from the artifact URIs in your input_dict, and often you might want to write your output to artifact URIs from your output_dict. This may include reading and writing to more than one split, as in the case of processing with train and eval splits.

from tfx.types import artifact_utils

train_input_examples_uri = artifact_utils.get_split_uri(
  input_dict['my_input_data'], 'train')
eval_input_examples_uri = artifact_utils.get_split_uri(
  input_dict['my_input_data'], 'eval')

train_output_examples_uri = artifact_utils.get_split_uri(
  output_dict['my_output_data'], 'train')
eval_output_examples_uri = artifact_utils.get_split_uri(
  output_dict[‘my_output_data'], 'eval')

In the example above the dictionary keys my_input_data and my_output_data are defined in the ComponentSpec for the component that you’re overriding the executor for.

class MyComponentSpec(tfx.types.ComponentSpec):
  PARAMETERS = {
      <...>
  }
  INPUTS = {
      'my_input_data':   
      ChannelParameter(type=standard_artifacts.Examples),
  }
  OUTPUTS = {
      'my_output_data':
      ChannelParameter(type=standard_artifacts.Examples),                                                            
  }

The splits are defined in the output Channel for the component that you’re overriding the executor for, typically in the constructor:

output_data = tfx.types.Channel(
  type=standard_artifacts.Examples,
    artifacts=[
      standard_artifacts.Examples(split=split)
      for split in artifact.DEFAULT_EXAMPLE_SPLITS
    ])
spec = LabelingComponentSpec(
  input_data=input_data,
  my_output_data=output_data)

Additional parameters are passed to your custom trainer executor using a custom_config dict. These can be retrieved by the custom executor using exec_properties.get(‘custom_config’).get(‘your_config_key’). In the example below, all of the additional arguments needed to submit a Google Cloud AI Platform training job can be found in _ai_platform_training_args.

_ai_platform_training_args = {
   'pythonModule': None,  # Will be populated by TFX
   'args': None,  # Will be populated by TFX
   'region': _gcp_region,
   'jobDir': os.path.join(_output_bucket, 'tmp'),
   'project': ‘your GCP project id’
}

...
trainer = Trainer(
    ...,
    custom_config={'ai_platform_training_args':
    _ai_platform_training_args})   
}

Linking the custom trainer’s output to the expected output artifact

Hooking up the custom trainer to emit the expected outputs is essential to the success of downstream components. For the Google Cloud AI Platform custom trainer, we serialize the executor input parameters so they can be transmitted as part of the GCP training job. Because the Google Cloud AI Platform (CAIP) executor is redirecting the default TFX executor to run on Google Cloud AI Platform, both take the same {transformed examples, a transform_fn, and a schema} input parameters to create a TF model. The custom executor used in this example submits a CAIP training job that will invoke (via run_executor.py) the default TFX trainer as the CAIP python module, effectively opening a conduit from the local workstation to run the TFX trainer on CAIP.

# Configure Google Cloud AI Platform job
    training_inputs = exec_properties.get('custom_config',
        {}).pop('ai_platform_training_args')
    executor_class_path = '%s.%s' % 
        (tfx_trainer_executor.Executor.__module__,
        tfx_trainer_executor.Executor.__name__)

    # Start Google Cloud AI Platform job
    return runner.start_cmle_training(input_dict, output_dict,
        exec_properties, executor_class_path, training_inputs)

Running the pipeline remotely with your custom executor

So far we’ve been assuming that your pipeline is running locally, using code available in your $PYTHONPATH. An upcoming blog post will explain how to execute custom executors packaged in containers, or as PyPI packages.

Related topics

In addition to Trainer, the TFX ExampleGen component also supports executor-level customization. ExampleGen provides a generic component and a base executor which apply ML best practices, e.g. data shuffling and consistent/configurable partitions.

If existing ExampleGen components don’t meet your needs, create a new Apache Beam PTransform for handling the conversion from an input split to TF examples and TFX will do the rest. The ExampleGen doc has more details.

For more information

To learn more about TFX check out the TFX website, join the TFX discussion group, and watch our TFX playlist on YouTube, and subscribe to the TensorFlow channel.

Creating a Custom TFX Executor was originally published in TensorFlow on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

OliveX is a Hong Kong-based company focused on fitness-related software, serving more than 2 million users since we first launched in 2018. Many of our users are elderly and our Baduanjin app helps them practice Baduanjin while minimizing the possibility of injury. To achieve that, we utilize the latest artificial intelligence technology in our app to automatically detect Baduanjin practicing moves and provide corresponding feedback to our users.

Goals and requirements

Baduanjin is a popular exercise that consists of eight kinds of limb movements and controlled breathing. Precise movements and control are critical for Baduanjin, which improves the health of the body both mentally and physically.

By using the “Smart Baduanjin” app, users can determine if they are performing the moves correctly by using AI to track their movements. By leveraging the latest machine learning technology, we hope to replace the traditional learning approach in which users simply follow an exercise video with a more enjoyable interactive experience in which users get feedback on their body movements in real time. We also hope that these features could help the elderly to practice Baduanjin more effectively and reduce the risk of injury.

After analyzing and prioritizing our goals, we defined the overall product requirements:

1. Our uses usually perform Baduanjin in an outdoor setting, so our product needs to be ‘mobile’
2. When a user practices Baduanjin, he/she usually needs to follow a demo video and imitate the movements from a coach, so our product should be able to play videos with sound
3. To provide valuable real time feedback to our users, we need to capture user body movements with the front-facing camera

Specifically regarding the body movement recognition, we want our algorithms to be able to do the following:

1. Recognize each Baduanjin movement in the overall sequence
2. Give scores based on the correctness of a user’s body movements
3. Provide correctional guidance on unsatisfactory movements

Technical analysis

ML framework selection

Based on the requirements above, we needed to select the right deep learning framework to implement our project. In particular, we were looking for a framework that has:

1. Strong support for mobile devices and can run smoothly, even on mid- to low-end smartphones
2. A friendly API design and rich debugging tools
3. Mature community support and resources available

After comprehensive investigation and evaluation, we found that TensorFlow Lite satisfied our needs and requirements. In addition, Google open sourced PoseNet, an app specifically designed for detecting human body poses, and provided demo code based on TensorFlow.js (EDITOR: we have recently released a PoseNet sample based on TensorFlow Lite). Google not only helped us finish the initial work for human body pose recognition with the help of the open source code, but also convinced us that our action recognition algorithm could work on mobile devices, since the performance on JavaScript was so good.

Algorithms

Body movement recognition

In the initial stage of our development, we investigated existing body movement recognition algorithms. Currently, mainstream algorithms are mostly based on analyzing sequential video frames. Although these algorithms could meet our requirements, the networks are fairly complicated and running inference on them consumes a lot of computational resources. But since one of our hard requirements was to run the model on mobile devices, we had to make a tradeoff between accuracy and performance.

Our approach was to first acquire key human body joints through PoseNet, and then recognize specific actions based on the sequence of body joint movement. Since PoseNet tracks only 17 body joints, the amount of computation was significantly reduced, as compared to the full size image. Below we show the workflow of our algorithm: first we used PoseNet to extract body joint data from input video, and then we carried out action classifications based on the body joint data.

Key actions

After deciding on the technical approach, we needed to define the key body movements important to recognize for the app. To do this, we transformed the body movement recognition problem to a typical machine learning classification problem. Below is a sample of how we defined the key Baduanjin movements:

We trained a traditional deep neural network to classify user body movements. After several iterations of hyperparameter tuning and training data optimization/augmentation, our final model had good accuracy and was able to satisfy our product requirements, as shown below.

Challenges on mobile devices

After finishing the deep learning model, our next step was to deploy our models on iOS and Android mobile devices. At first, we tried TensorFlow Mobile. But since we needed to get recognition results in real time, TensorFlow Mobile was not a viable option since its performance did not meet this requirement.

As we were trying to solve the performance challenge, Google released TensorFlow Lite, which was a big leap from TensorFlow Mobile in terms of performance. We compare the two offerings below:

We also show the initial benchmark results on our model below:

Based on the benchmark numbers, we concluded that it wasn’t feasible to do body movement recognition in real time on most Android devices based on a 512x512 input size. To solve this problem, we profiled our machine learning models and found that PoseNet was the bottleneck, which consumed 95% of the computational time. So we adjusted PoseNet input size and hyperparameters, and retrained our action recognition algorithm to compensate for the accuracy loss introduced by reduced input size. In the end, we chose 337x337 RGB as the input and 0.5 as the width multiplier for MobileNet on Android.

Our target users are mostly elderly, so they tend to have lower-end devices. Even though we improved the performance by adjusting PoseNet parameters, it was still not satisfactory. Therefore, we resorted to the ubiquitous accelerator in smartphones: GPU. Google happened to release TensorFlow Lite GPU delegate (experimental) around that time, which saved us plenty of engineering resources.

Mobile GPU considerably accelerated our model execution. Below is our benchmark on several popular devices among our users, with and without GPU delegate.

Since Baduanjin moves are relatively slow for older users, our product could run smoothly on most devices after integrating TensorFlow Lite GPU delegate (experimental).

Conclusion

We successfully finished our Smart Baduanjin product on both iOS and Android, and have received positive feedback from our testing users. By leveraging ML technology and TensorFlow, we provided a ‘tutorial model’ for Baduanjin beginners so that they can follow the demo video and learn the moves. For experienced Baduanjin practitioners, we provided valuable feedback such as scores to help them further improve their skills. Currently, ‘Smart Baduanjin’ is available for free in both the App Store and Google Play.

Meanwhile, OliveX is actively exploring applying human body pose estimation to other fitness exercises. We found that many other exercise practices are just like Baduanjin, in that the correctness of the practitioners’ body movements matters a lot. Correct body movements not only help people avoid bodily injury, but also increase their exercise efficiency. Thus, we want to transfer the knowledge and skills we obtained from the Baduanjin project to other domains as well.

Body movement recognition in the ‘Smart Baduanjin’ App was originally published in TensorFlow on Medium, where people are continuing the conversation by highlighting and responding to this story.