I was lucky to participate in the PAISS summer school organized by Inria Grenoble and Naver Labs Europe in France. During one week, several Deep Learning stars shared their latest research.

In this blog post, my goal is to give AI enthusiasts & researchers who missed the summer school the opportunity to see what has been discussed, provide a few details about the ongoing projects/advances in the different fields of Deep Learning and references to the must-read papers in case you want to explore by yourself what I quickly introduce. I will not be able to sum up 40 hours of conference, instead I will focus on what particularly interested me.

Most of the slides are now available: https://project.inria.fr/paiss/program/

1. Deep Learning explained with metaphors
2. Visual Search
3. Weak and Unsupervised Learning for Image Analysis
4. Video Understanding
5. Natural Language Processing
6. Theoretical Analysis
7. On the road to Artificial General Intelligence

Let’s go

1. Deep Learning explained with metaphors

Yann Lecun is the king, among other things, of metaphors: here is a non exhaustive list of its favorite. If you often need to explain Deep Learning basic principles, these metaphors may help you.

• The “classical” cake of Yann Lecun

• Back propagation as a tuning of radio potentiometers

You can compare weights of your network with the potentiometers of an old radio: a step of backpropagation when trying to output the good label is similar to a slight change of the potentiometers when trying to have a good output frequency.

• Gradient Descent as a descent in the fog

Imagine you are at in a foggy mountain, and you try to reach the nearest village. You can only see few meters away, but you know that the village is at low altitude. A good heuristic would be to see around you the steepest slope and continue iteratively. This is exactly what Gradient Descent does.

2. Visual Search

Given an image, you somehow compute an embedding and find the most similar item inside a catalog with a basic dot product. The one million dollar question is: how do you extract these features ?

A. Basic Image Features Extraction

Some “old fashioned” methods used low level cues (color histogram, textures, shape, …) or local quantized descriptors (SIFT, SURF [1]). Finally techniques such as Fisher Vector [2] or Vlad aggregate local descriptors to obtain global descriptors.

But that was before the deep learning success. Now a convolutional neural network trained on IMAGENET is the most frequently used features extractor. Note that hybrid methods have been tested by adding a Vlad layer [3, 4] or a Fisher Layer at the end of a CNN.

[1] Karami, E., Prasad, S., & Shehata, M. (2017). Image matching using SIFT, SURF, BRIEF and ORB: performance comparison for distorted images. arXiv preprint arXiv:1710.02726.

[2] Perronnin, F., Sánchez, J., & Mensink, T. (2010, September). Improving the fisher kernel for large-scale image classification. In European conference on computer vision (pp. 143–156). Springer, Berlin, Heidelberg.

[3] Delhumeau, J., Gosselin, P. H., Jégou, H., & Pérez, P. (2013, October). Revisiting the VLAD image representation. In Proceedings of the 21st ACM international conference on Multimedia (pp. 653–656). ACM.

[4] Arandjelovic, R., Gronat, P., Torii, A., Pajdla, T., & Sivic, J. (2016). NetVLAD: CNN architecture for weakly supervised place recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 5297–5307).

B. How to do better than IMAGENET embeddings ?

An end-to-end methodology to “distinguish between particular objects, even if they belong to the same semantic class” (inspired from [5]) would be:

• create a dataset by google queries (in order to have several photos of the same object) and clean it automatically by discarding the large amount of mislabeled images using old fashioned methods (described earlier) in an offline mode, while keeping matching boxes
• “regional maximum activations of convolutions” (RMAC) features [6] with pooling over several regions and where features are normalized, shifted/projected and then normalized again

• learn how to choose these regions with Region Proposal Network like in Faster RCNN [7]
• fix the alignment issue with Spatial Transformer Network [8]
• do not use a classification loss but a ranking loss (such as the triplet loss) with curriculum learning and hard negative mining
• embedding compression with product quantization for faster query
• iteratively leverage the topology of the catalog dataset with query expansion/diffusion
• handle high resolution (or multi-resolution), data augmentation and keep the images undistorted

If you want to make “semantic” identification, you can benefit from weak information inside the text. That’s why in [9], they use multimodal dataset and try to reconstruct the human captions as a proxy for image embedding.

[5] Gordo, A., Almazan, J., Revaud, J., & Larlus, D. (2017). End-to-end learning of deep visual representations for image retrieval. International Journal of Computer Vision, 124(2), 237–254.

[6] Paulin, M., Douze, M., Harchaoui, Z., Mairal, J., Perronin, F., & Schmid, C. (2015). Local convolutional features with unsupervised training for image retrieval. In Proceedings of the IEEE international conference on computer vision (pp. 91–99).

[7] Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems (pp. 91–99).

[8] Jaderberg, M., Simonyan, K., & Zisserman, A. (2015). Spatial transformer networks. In Advances in neural information processing systems (pp. 2017–2025).

[9] Gordo, A., & Larlus, D. (2017, July). Beyond instance-level image retrieval: Leveraging captions to learn a global visual representation for semantic retrieval. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

3. Weak and Unsupervised Learning for Image Analysis

As labellisation is the bottleneck of most deep learning techniques, we should aim at the “Genoise” of Lecun ‘s cake

A. Benefit from Internet’s data

Internet is full of weak labels, collected automatically without human annotations, such as metadata for image recognition.

Indeed, only image level labels are needed for cosegmentation [10] to separate objects from background. Similarly, [11] learns to name character in video thanks to the weak labels given by the script of the video in the TV show Buffy while [12] exploits temporal constraints.

Furthermore, at Heuritech we published [13] where we learn a multimodal embedding of text and image by matching e-commerce image with their description. More recently [14] takes weak supervision to a whole new level, and learns to predict hashtags on 3.5 billion Instagram images.

[10] Joulin, A., Bach, F., & Ponce, J. (2012, June). Multi-class cosegmentation. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on (pp. 542–549). IEEE.

[11] Sivic, J., Everingham, M., & Zisserman, A. (2009, June). “Who are you?”-Learning person specific classifiers from video. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on (pp. 1145–1152). IEEE.

[12] Bojanowski, P., Lajugie, R., Bach, F., Laptev, I., Ponce, J., Schmid, C., & Sivic, J. (2014, September). Weakly supervised action labeling in videos under ordering constraints. In European Conference on Computer Vision (pp. 628–643). Springer, Cham.

[13] Corbiere, C., Ben-Younes, H., Ramé, A., & Ollion, C. (2017, October). Leveraging weakly annotated data for fashion image retrieval and label prediction. In Computer Vision Workshop (ICCVW), 2017 IEEE International Conference on (pp. 2268–2274). IEEE.

[14] Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., … & van der Maaten, L. (2018). Exploring the Limits of Weakly Supervised Pretraining. arXiv preprint arXiv:1805.00932.

B. Self-supervised learning

It’s a form of unsupervised learning where the data provides the supervision: in general, you withhold some part of the data, and task the network with predicting it. The task defines a proxy loss, and the network is forced to learn what we really care about, e.g. a semantic representation, in order to solve it. The challenge is to find the most meaningful proxy task

If you have a dataset of images, you can try to learn:

• relative positioning: given 2 patches of an image, try to know how they are positioned relatively
• colorization: try to find the RGB colors of a grey image
• exemplar networks: learn to match different data augmentations of the same image
• image transformations: which image has the correct rotation [15]

When you have a dataset of videos, you can use the temporal information:

• is this sequence of frames real or fake? [16]
• is the video going forward or backward ? [17]
• try to colorize the video given the first colorized frame by learning soft assignment from frame to the original frame by implicit tracking

It becomes really funny when we use the implicit constraint of video and sound:

• is the sound appropriate or not with the video? [18], with triplet loss
• is the sound synchronized or not with the video ? [19]

[15] Gidaris, S., Singh, P., & Komodakis, N. (2018). Unsupervised Representation Learning by Predicting Image Rotations. arXiv preprint arXiv:1803.07728.

[16] Misra, I., Zitnick, C. L., & Hebert, M. (2016, October). Shuffle and learn: unsupervised learning using temporal order verification. In European Conference on Computer Vision (pp. 527–544). Springer, Cham.

[17] Wei, D., Lim, J. J., Zisserman, A., & Freeman, W. T. (2018). Learning and Using the Arrow of Time. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(pp. 8052–8060).

[18] Arandjelović, R., & Zisserman, A. (2017). Objects that sound. arXiv preprint arXiv:1712.06651.

[19] Chung, J. S., & Zisserman, A. (2016, November). Out of time: automated lip sync in the wild. In Asian Conference on Computer Vision (pp. 251–263). Springer, Cham.

C. Unsupervised Features Extraction

Some researchers, such as Joulin, even aim at learning images features with no supervision at all.

• [20] fixes a set of target representations and constrain the deep features to align to them. This domain agnostic approach avoids the standard unsupervised learning issues of trivial solutions and collapsing of features.
• DeepCluster [21] is a clustering method that jointly learns the parameters of a neural network and the cluster assignments of the resulting features.

[20] Bojanowski, P., & Joulin, A. (2017). Unsupervised learning by predicting noise. arXiv preprint arXiv:1704.05310.

[21] Caron, M., Bojanowski, P., Joulin, A., & Douze, M. (2018). Deep Clustering for Unsupervised Learning of Visual Features. arXiv preprint arXiv:1807.05520.

4. Video Understanding

A. Optical Flow

Cordelia Schmid introduced optical flow as the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer and a scene.

A solution to the optical flow problem was proposed in the Lucas Kanade algorithm with assumptions such as brightness consistency, motion consistency, small motion, optical flow consistency for nearby points and with constraints on gradient.

Recently, U-Net networks was used to predict optical flow in Flownet2 [22] and FlyingThings3D [23]. Interestingly, they initialize their network with synthetic data and then fine tune with a custom small dataset.

[22] Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017, July). Flownet 2.0: Evolution of optical flow estimation with deep networks. In IEEE conference on computer vision and pattern recognition (CVPR) (Vol. 2, p. 6).

[23] Mayer, N., Ilg, E., Hausser, P., Fischer, P., Cremers, D., Dosovitskiy, A., & Brox, T. (2016). A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4040–4048).

B. Action Recognition

The goal is to identify the main action of a video, using RGB frames along with optical flows (computed with Flownet) as inputs.

Zisserman [24] studied a number of ways of fusing ConvNet towers both spatially (RGB with 3 channels) and temporally (externally computed optical flows inputs with 2 channels) in order to best take advantage of this spatio-temporal information. [25] got better results with 3D ConvNet.

[26] made a review of all methods and introduce a new Two-Stream Inflated 3D ConvNet (I3D) that is based on 2D ConvNet inflation: filters and pooling kernels of very deep image classification ConvNets are expanded into 3D, making it possible to learn seamless spatio-temporal feature extractors from video while leveraging successful ImageNet architecture designs and even their parameters. Performances are judged according to the kinetics human action video dataset [27], with 400 human action classes and over 400 clips per class collected from realistic, challenging YouTube videos.

[24] Feichtenhofer, C., Pinz, A., & Zisserman, A. (2016). Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1933–1941).

[25] Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015). Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision (pp. 4489–4497).

[26] Carreira, J., & Zisserman, A. (2017, July). Quo vadis, action recognition? a new model and the kinetics dataset. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on (pp. 4724–4733). IEEE.

[27] Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., … & Suleyman, M. (2017). The kinetics human action video dataset. arXiv preprint arXiv:1705.06950.

C. Video Tracking

The goal is to track an object in different consecutive frames.

[28] introduces a motion RPN (from optical flows) which generates high-quality proposals and is complementary to the appearance RPN (from RGB), and then link frame-level detections with the Viterbi algorithm.

But the best networks learn everything end to end and leverage the temporal continuity of videos instead of operating at the frame level. As input [29, 30] take a sequence of frames and output sequences of bounding boxes with associated scores. Inspired from [28] and [26], [31] localizes atomic visual actions, and handles multiple annotations for each person in the video.

[28] Peng, X., & Schmid, C. (2016, October). Multi-region two-stream R-CNN for action detection. In European Conference on Computer Vision (pp. 744–759). Springer, Cham.

[29] Kalogeiton, V., Weinzaepfel, P., Ferrari, V., & Schmid, C. (2017). Action tubelet detector for spatio-temporal action localization. ICCV, Oct, 2.

[30] Hou, R., Chen, C., & Shah, M. (2017, October). Tube convolutional neural network (T-CNN) for action detection in videos. In IEEE international conference on computer vision.

[31] Gu, C., Sun, C., Ross, D. A., Vondrick, C., Pantofaru, C., Li, Y., … & Schmid, C. (2017). AVA: A video dataset of spatio-temporally localized atomic visual actions. CoRR, abs/1705.08421, 4

5. Natural Language Processing

A. Machine Reading with attention

The Stanford Question Answering Dataset is a set of 10,000 questions pertaining to some 500 Wikipedia articles. The answer to each question is a particular span of text from the corresponding piece of writing.

Whereas CNN has been the cornerstone of latest success in Image recognition, attention models had lot of success recently for NLP in machine reading. The key idea is to get embeddings for the question and the elements in the documents (using GRU or word embeddings), them merge them with attention, in order to have a question aware representation, and output the final answer.

Naver Labs, led by Julien Perez, made an impressive demo where given a question, their model is able to find the good span of text in a set of documents [32]. With memory network, you can also perform reasoning [33].

[32] Kadlec, R., Schmid, M., Bajgar, O., & Kleindienst, J. (2016). Text understanding with the attention sum reader network. arXiv preprint arXiv:1603.01547.

[33] Liu, F., & Perez, J. (2017). Gated end-to-end memory networks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers (Vol. 1, pp. 1–10).

B. New breakthroughs in Machine Translation

Like in many other area, old fashioned methods (Statistical Machine Translation) are now beaten by open source purely deep learning techniques. For example, Kyunghyun Cho adverts the FAIR library FairSeq in Pytorch.

Interesting new ideas raise my attention:

• [34] learns to translate in real time: the decoder does not need to wait the final output of the encoder to begin to translate
• [35] uses purely attention without GRU
• learning of an universal encoder decoder for multilingual translation
• [36] does not even use parallel data for translation in a purely unsupervised way
• [37] finds the best architecture with meta learning

[34] Gu, J., Neubig, G., Cho, K., & Li, V. O. (2016). Learning to translate in real-time with neural machine translation. arXiv preprint arXiv:1610.00388.

[35] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., … & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems(pp. 5998–6008).

[36] Lample, G., Denoyer, L., & Ranzato, M. A. (2017). Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.00043.

[37] Zoph, B., & Le, Q. V. (2016). Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578.

C. Biologically inspired Speech to Text

Speech to text is now present in our everyday life, with tools such as Siri/Alexa. First method got inspired by the biological process of the human ear, consequence of generations of natural selection. Our ear makes frequency compression, dynamic range compression, short term compression, and contrast enhancement. The scientists adapted it to time windowing, fourier transform, mel filterbank, log scaling, and discrete cosine transform. But now with deep learning (and GRU), no need to such handmade pipelines [38].

[38] Graves, A., Mohamed, A. R., & Hinton, G. (2013, May). Speech recognition with deep recurrent neural networks. In Acoustics, speech and signal processing (icassp), 2013 ieee international conference on (pp. 6645–6649). IEEE.

6. Theoretical Analysis

A. Deep Learning in Non Euclidean Space

Convolutional Layers enabled deep learning to make huge breakthroughs in 1D, 2D, or 3D euclidean spaces. But we can generalize CNN architectures to non-Euclidean domains (graphs and manifolds) with geometric deep learning. The basic idea is to define a convolution as the element wise product of the Fourier Transforms of two signals (the convolution theorem) [39]. You can learn more about it in: http://geometricdeeplearning.com/

[39] Bronstein, M. M., Bruna, J., LeCun, Y., Szlam, A., & Vandergheynst, P. (2017). Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4), 18–42.

B. Theoretical Optimization Guaranteed magically transferred to Deep Learning

When trying to solve a machine learning problem, our loss is composed of the sum of 3 terms

• we minimize over a class of functions and not over all possible functions: approximation error
• we minimize over our training set and not the true distribution: estimation error
• the optimization process may not reach the global minimum: optimization error

The goal of optimization researchers is to reduce as quick as possible and as much as possible the optimization error. But as machine learning researchers, we are less interested by how the training loss decreases than by how a model generalizes over new samples. [40] develops a theoretical framework that takes into account the effect of approximate optimization on learning algorithms, and proves that there is no need to optimize below a statistical error. So you can stop your learning process when the loss is close to 0.

From a theoretical point of view, given a function that is smooth (its gradient is L lipschitz) and convex, Gradient Descent has a convergence rate in 1/nb_step if we have a perfect learning rate of 1/L. L can be computed (with a line search) because the largest value of the Hessian matrix of the loss function is an upper bound of L. If you add a regularization function that is not smooth, it still works with a proximal gradient [41]. Momentum such as Nesterov improves the convergence rate.

But as complete computation of the gradient is intractable, we often use Stochastic Gradient Descent with a theoretical convergence rate in 1/ sqrt(nb_step) [42]. It is even quicker with less parameters tuning when using better estimation of the gradient like in Stochastic Average Gradient [43]: at each step, you refine the gradient of the corresponding training sample and then sum over all training samples.

But in Deep Learning, the loss is not even convex! fortunately for us, everything works fine. Here are some common tricks to get good optimization processes [44]:

• reduce the learning rate by 10 whenever your loss is not decreasing anymore
• use small batch size (Yann Lecun made it very clear in [45] that the only reasons we do not use batch size 1 are hardware limitation) even if it’s controversial
• optimizer with momentum works, maybe because it benefits from local convexity, convexity along trajectories or escape more easily from saddle points
• use adaptive learning rate (like Adam [46])

[40] Bottou, L., & Bousquet, O. (2008). The tradeoffs of large scale learning. In Advances in neural information processing systems (pp. 161–168).

[41] Schmidt, M., Roux, N. L., & Bach, F. R. (2011). Convergence rates of inexact proximal-gradient methods for convex optimization. In Advances in neural information processing systems (pp. 1458–1466)..

[42] Moulines, E., & Bach, F. R. (2011). Non-asymptotic analysis of stochastic approximation algorithms for machine learning. In Advances in Neural Information Processing Systems (pp. 451–459).

[43] Schmidt, M., Le Roux, N., & Bach, F. (2017). Minimizing finite sums with the stochastic average gradient. Mathematical Programming, 162(1–2), 83–112.

[44] Bottou, L. (2012). Stochastic gradient descent tricks. In Neural networks: Tricks of the trade (pp. 421–436). Springer, Berlin, Heidelberg.

[45] Masters, D., & Luschi, C. (2018). Revisiting Small Batch Training for Deep Neural Networks. arXiv preprint arXiv:1804.07612.

[46] Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

C. Correlation is not Causation

Leon Bottou stated that our networks are not good enough at reasoning and do not use structure. That’s why, most of our algorithms are not able to generalize to images in the wild and rely way too much on correlation rather than on causation.

7. On the road to Artificial General Intelligence

A. Meta Learning: Learning to Learn

While the goal of few shot learning is to design an algorithm that generalizes well on small dataset, meta learning aims at directly learning this algorithm. We will have a meta train set that is composed of several train-test datasets on which we learn the meta algorithm (the loss being the one of the test datasets), and a meta test set that is composed of other train/test datasets on which we apply our meta learning algorithm. The overall test loss is the average loss on the test datasets of the meta test datasets.

To define a meta learner, you can take inspiration from a known learning algorithm: [47] is similar to Nearest Neighbours, [48] to Bayesian classifier or [49] to gradient descent. Or you can derive it from a black box neural network, such as [50] and [51]

[47] Vinyals, O., Blundell, C., Lillicrap, T., & Wierstra, D. (2016). Matching networks for one shot learning. In Advances in Neural Information Processing Systems (pp. 3630–3638).

[48] Snell, J., Swersky, K., & Zemel, R. (2017). Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems (pp. 4077–4087).

[49] Ravi, S., & Larochelle, H. (2016). Optimization as a model for few-shot learning.

[50] Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., & Lillicrap, T. (2016). One-shot learning with memory-augmented neural networks. arXiv preprint arXiv:1605.06065.

[51] Mishra, N., Rohaninejad, M., Chen, X., & Abbeel, P. (2018). A simple neural attentive meta-learner.

B. Deep Reinforcement Learning

Reinforcement Learning refers to a kind of Machine Learning method in which the agent receives a delayed reward in the next time step to evaluate its previous action (see [52] for an overall introduction). Remi Munos gave an inspiring speech about the spirit you should have when doing research in RL (or any science in fact): when facing a need, his goal is to find an idea that will have a theoretical benefit in a simplified setting and that gives numerical performance on real data.

He was facing the limitations of traditional Reinforcement Learning algorithms.

1. off-policy gradient such as Deep Q Network (DQN [53])

• pros: any kind of exploration schema, dissociation from acting with learning, prioritized experience replay
• cons: 1 step bootstrapping, slow to propagate information, accumulate errors, no RNN

2. on-policy gradient such as Asynchronous Advantage Actor-Critic (A3C [54])

• pros: multi steps training (fast propagation of information), asynchronous (parallelization), possible use of RNN
• cons: no memory replay, no exploration

We would like to enjoy the benefits from dissociating acting and learning from DQN while being able to do multi steps rewards like A3C. He does so by defining a behavior policy that is different from its target policy, and using an importance sampling weighted schema that gives an unbiased estimate of the Value Function: they reweight the traces of the trajectory by a smart ratio that maintains stability while being efficient. The behavior policy will enable the exploration of new states: if it ends up being a good idea, the value of the previous states will be updated accordingly, but otherwise they will not decrease too much.

This theoretical approach called RETRACE [55] enabled several new architecture at Deepmind such as REACTOR [56], and was further improved in V-TRACE for IMPALA [57].

[52] Li, Y. (2017). Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274.

[53] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., … & Petersen, S. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529.

[54] Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., … & Kavukcuoglu, K. (2016, June). Asynchronous methods for deep reinforcement learning. In International conference on machine learning (pp. 1928–1937).

[55] Munos, R., Stepleton, T., Harutyunyan, A., & Bellemare, M. (2016). Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems (pp. 1054–1062).

[56] Gruslys, A., Azar, M. G., Bellemare, M. G., & Munos, R. (2017). The Reactor: A sample-efficient actor-critic architecture. arXiv preprint arXiv:1704.04651.

[57] Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., … & Legg, S. (2018). IMPALA: Scalable distributed Deep-RL with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561.

C. Robots are hard to train

Martial Hebert clearly states that whereas you can be satisfied with a 95 % accuracy for image reasoning, robots can not fail ! they need a huge level of reliability. This is highly complicated because a robot needs perception, interpretation and even reasoning.

The idea is to have an explicit representation of the failure that could monitor the system: it would rather do nothing than a mistake. During training, we reweight our training dataset similarly to boosting, train several predictors, and at test time an adaptive planning networks chooses which predictor to use. It becomes really meta when you need a meta planner to select which planner to use (to select with predictor to use).

This algorithm also needs to be anytime algorithm [58], meaning that they should respect

• interruptibility: give answer at any time
• monotonicity: answers do not get worse over time
• diminishing returns: initial improvements are more useful

Nicolas Mansard shows we have hard time transferring breakthroughs in Deep Learning to Motion generation for robots, because the space is too big to explore with DRL and that we need to give some prior to mechanics, otherwise it fails miserably. In conclusion, it is really difficult to use Reinforcement Learning for Robots: research is still in progress ! Remember that Pure RL is hard to use in the real world, because anything you do in the real world can kill you and that you can’t run the real world faster than real time.

[58] Zilberstein, S. (1996). Using anytime algorithms in intelligent systems. AI magazine, 17(3), 73.

Conclusion

You may have noticed that a general pattern emerged: experimental Deep Learning is replacing theoretical old fashioned methods. Even if theoretical verification would be much appreciated, the progress made recently in very different domains are absolutely amazing!

Here at Heuritech https://www.heuritech.com/ we use plenty of these methods in our R&D team when developing our visual recognition system for real world images. We empower fashion teams to monitor in real time both products and trends through cutting-edge image recognition technology applied on millions of social media images every day.

Join the fun!

Alexandre Ramé