Tes chroniques m’inspirent, Matthieu, alors je t’écris pour passer le temps en attendant que le now serving progresse du 85 sur lequel il semble stallé jusqu’au 05 que je tiens entre mes doigts. J’attends qu’un commis puisse me répondre et m’assister dans la très délicate opération qui consiste à déclarer que mon permis de conduire a été volé, que je lui montre ma carte d’assurance maladie, et qu’il me charge 13$ avant de me renvoyer chez moi. Une opération que j’aurais très bien pu faire de la maison, même sans intelligence artificielle. Je me dis d’ailleurs que, si une bonne partie de notre société ne semble pas encore rendue à l’ère du web, on peut fantasmer sur la révolution IA, il risque de se passer un peu de temps avant qu’on la vive poulevrai.

Bref, long préambule pour parler créativité et IA. Ton passage par les statistiques dans ta chronique est fort approprié. En fait le machine learning c’est pas mal juste ça : réussir à capturer, à l’aide des millions de paramètres d’un réseau de neurones, la façon dont les données sont distribuées. On peut rendre ça plus concret en imaginant un paquet de points qui flottent dans l’espace. Le machine learning génératif nous permet de tracer des courbes de niveau qui capturent les endroits où la densité des points est plus grande. Ces courbes de niveau permettent de répondre à la question “si un nouveau point apparaissait, où aurait-il le plus de chances de se trouver?”

Bon, en réalité l’espace où se trouve les points n’est pas l’espace 3D auquel on est habitués, mais un hyperespace qui compte des milliers de dimensions. Bonne chance pour visualiser ça… Mais l’idée reste la même : réussir à trouver des courbes de niveau qui permettent de générer de nouveaux points dans des régions de cet hyperespace où on a une bonne probabilité de trouver un point.

Chacun de ces points peut encoder une image, un poème, une pièce de musique ou un film… Mais les fameuses courbes de niveau n’existent que dans les régions de l’hyperespace où se trouvent déjà des points… C’est-à-dire dans les régions qui encodent des images, des poèmes ou des films qu’on a fournit comme points de donnée.

Les algos de ML peuvent synthétiser de nouveaux points (donc de nouvelles images, de nouveaux poèmes ou de nouveaux films) mais ceux-ci sont, dans une certaine mesure, condamnés à ressembler aux données fournies. Cette ressemblance est complexe et si on demande à un expert de l’expliquer il parlera probablement d’une “esthétique similaire à…”, ou d’une “trame sonore qui rappelle…” ou encore “d’un récit dans le style de…” Mais, si l’algorithme fait bien son travail, le résultat sera nécessairement sans surprise. On ne verra jamais une “esthétique similaire aux teletubbies avec une trame sonore qui rappelle les documentaires de National Geographic et un récit dans le style des frères Cohen” parce que l’algorithme n’a observé aucun point dans la région de l’hyperespace qui encoderait ce genre de film.

Maintenant, la créativité humaine c’est justement cette capacité qu’on a à s’aventurer dans des régions inexplorées de l’hyperespace mais qui parlent à un certain public, des œuvres surprenantes pour lesquelles certaines personnes sont “mûres”.

Une œuvre d’art surprenante est celle qu’un certain public comprend mais qu’il n’avait jamais imaginée auparavant.

Pour nos algos de ML, l’histoire de la créativité humaine c’est l’histoire de courbes de niveaux qui s’étirent et se déforment pour tracer des patrons de plus en plus complexes dans l’hyperespace… Mais on est très loin de pouvoir créer un algorithme qui saura modéliser les humains d’une manière qui lui permettrait de prédire quelle courbe de niveau devrait être étirée et déformée pour saisir les images, les poèmes ou les films pour lesquels les humains sont mûrs.

La créativité humaine a donc de beaux jours devant elle. Peut-être même que le Prométhée de ta nouvelle nous aidera à mieux comprendre la spécificité de notre créativité. Peut-être qu’il sera une source de motivation pour les artistes en leur offrant une carte de l’hyperespace connu dans lequel on verra d’immenses régions inexplorées…

Do you realize that tools as useful as Google or Facebook are almost entirely paid for through ads?

It sounds innocuous if we think of modern ads as a simple evolution of the TV ads we used to laugh at when we still watched TV. But TV ads are to modern web-based ads as water guns are to bazookas. If we regulate them as water guns we’ll be in trouble soon.

I think part of the problem is that, on an individual level, an ad-based model makes a lot of sense. Each one of us, individually, think that we’re influenced very little by ads. But on a larger scale science doesn’t lie: ads work.

Advertisements are a mechanism that allows money to influence the way a population thinks.

Funding our most valuable tools with ad-based mechanisms is a social problem, in the same way the current environmental crisis is a social problem. And as with the environmental crisis, we will not solve that problem by convincing people to pay individually for their most useful tools. On an individual level, the cost is too high…

So, like the environmental crisis I believe the solution will not appear if we only rely on “the free market” to find it. We will need to get together and use our creativity. We will need to find ways for these great tools to maintain their usefulness without having to pay for them by accepting to live in a less democratic society.

I’m not sure what the solution is, but digging deeper in the ways in which our instantaneous reactions to ads differ from our deeper aspirations is likely a part of it. That’s a subject I love, and for which I believe AI, which created the problem in the first place, may also hold part of the solution. I explore it in this (relatively old) medium post.

Every corporation that has successfully deployed artificial intelligence solutions has done so by installing a very deliberate internal culture of sharing data. Take Google’s example. In 2011, the Mountain View giant had more than seventy different privacy policies for their various products. The following year, they aggregated most of these into a new main privacy policy, explaining what information was collected and how it was being used by all Google products.

It’s no coincidence that this move took place in 2012, the year deep learning really started picking up. Back then, every AI researcher was discovering the following adage:

The greatest killer of AI is siloed data.

That’s because deep learning, the key technology behind the current AI revolution, is known to be very data hungry. By keeping the data for each of their products siloed, Google couldn’t unleash deep learning on the aggregate of their rich datasets, which was a problem.

Diversity matters

Breaking down silos doesn’t only increase the amount of data on which AI can be trained, it also increases the diversity of this data. Different products perceive users in different ways, and these different viewpoints open up a world of possibilities. For example, a clever engineer may combine the datasets from Google Fit and Google Music to learn which music is best suited to different activities.

AI can do the same, only better. Where humans can find a few connections, an AI can identify thousands of them. A machine learning system let loose on these combined datasets would therefore be able to discover many unforeseen correlations that have the ability to improve each of these products.

The curse of secrecy

Given that huge and diverse datasets are required to train modern machine learning systems, how can corporations with smaller or more specialized datasets benefit? The prospects are grim. Indeed, an analysis of the Big Data business model, in which corporations allow their data to be collected by huge aggregators, shows that this approach mostly benefits giant tech companies.

Total data secrecy might sound more appealing, yet this is also a recipe for failure. A corporation building a wall between its data and the rest of the world would also be blocking out the many benefits of deep learning.

Representation learning to the rescue

Fortunately, progress in the field of deep learning research hints of a future where every corporation could benefit from AI. This research direction has a name: “representation learning”.

Representation learning is anchored in deep learning’s ability to understand data on different levels. Let’s take a quick look at how this happens.

Deep learning relies on neural networks, which are systems made of layered units. Each of these units, called perceptrons, are very simple things. They receive a bunch of input signals, sum them up in a weighted fashion, and if the sum is large enough, they fire a signal to the perceptrons on the next layer.

Once built, neural networks need to be trained on a specific task, for example determining if an image contains a cat. Training ensures that each perceptron learns the weights to apply to the signals coming from their friends in the previous layer.

We’ve become really good at training neural networks in that we can easily create a network where the perceptrons on the final layer say “yes” only when an image contains a cat. What we’re not so good at, however, is understanding and controlling what happens between the input and the output of this network.

Learning a tangled mess

Let’s look at our cat detector a little more closely. In this type of neural network, the input to the perceptrons on the very first layer is image pixels. After training, an analysis of the perceptrons on that layer shows that they fire when a region of the image contains edges at different angles. The first layer of the neural network has learned to detect object boundaries!

Good job first layer! Now let’s try to guess what the deeper layers have learned. Maybe the second one is able to identify a combination of two lines (i.e. a corner); perceptrons on the third layer might fire when they spot a frequent shape like a circle or a rectangle; deeper still, we may find perceptrons that are able to identify an eye or a nose.

In practice, though, these deeper layers rarely learn to detect such neatly categorized components. Their outputs correlate with the content of the image in a way that’s seemingly random. It all gets magically sorted out somehow, and the network still performs well on the task it was trained for. However, unless we provide some extra details, the components that are detected by deeper layers do not seem to carry any meaning for us poor humans.[1]

Worse, components detected by deeper layers are not necessarily meaningful for other neural networks trained on different tasks. In machine learning parlance, we say that perceptrons on deeper layers are “entangled”, hinting at the fact that meaningful components are somehow encoded by the output of multiple perceptrons in a way that cannot be easily teased out.

From a technical standpoint, a deep neural network is not a “black box”, because we can see which perceptrons are firing at any time. Yet our comprehension of these systems is limited, as we don’t really understand what it means for most of the perceptrons to fire. By learning to detect meaningful components, representation learning will therefore help us understand not only what goes on in the first layers (the edge detectors), or on the last layer (the cat detector), but everywhere in between.

It’s still challenging to design networks and algorithms that learn good representations. In fact, it’s a very active field of research and every year the International Conference on Learning Representation (ICLR) examines new breakthroughs. If you’re interested in the nitty-gritty details, this paper by Bengio et al. is a good starting point.

Better representations mean better understanding

A system that can learn meaningful components at every level is a system that has a better understanding of the task it’s tackling. This pays off in a number of ways:

• Understandable machine learning. The system is more understandable to a human analyzing its internal behaviour.
• Domain adaptation. The system can be trained on different but related tasks using a smaller amount of data.
• Transfer learning. The system can share its understanding with other systems tackling tasks in which similar components are important.

In short, advances in representation learning make it possible for a trained neural network to share its expertise with humans as well as with other deep learning systems.

Sharing expertise is the secret of efficient systems

Machines that can abstract meaningful components and share this high-level expertise with other systems is quite a novel idea. Yet, humans and corporations have been sharing expertise for a very long time.

Take the example of Silicon Valley, arguably the most fruitful technological ecosystem in the world. Its success is largely due to the fact that engineers and researchers bounce around from one great company to another. Each time, these talented individuals get exposed to the inner workings of a new company and, as a result, improve their skills and knowledge in the process.

NDAs may prevent an ex-employee from sharing data they’ve been exposed to, but the expertise they gained stays with them. As a result, everyone benefits, and no single big “expertise aggregator” comes out on top.

An ecosystem of AI systems

In the same way that successful corporations seek out ecosystems in which they can find talented employees, they should strive to build their AI in an environment that gets them exposed to as many different tasks as possible.

Representation learning opens the door to AI systems that can improve their expertise over time as they get exposed to different tasks. This, in turn, leads to systems that can perform significantly better on a totally new task, even when this task only comes with a small and specialized dataset.

In time, advances in representation learning will lead to greater exchanges of expertise between AI systems and the old adage will change to:

The greatest killer of AI is siloed expertise.

At this point we will have truly achieved the dream of democratized AI, allowing every corporation — large or small, big tech or not — to benefit from the power of machine learning.

[1] This is a bit of an oversimplification. With some work, it is possible to find a few perceptrons in lower levels that detect components like “faces” or “wrinkles” or “text in images”. It remains hard to explain what most perceptrons do, though, and why the network still needs them to emit its final answer.

Illustrations by Morgan Guegan

The future of AI: from siloed data to shared expertise was originally published in Element AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Element AI has just turned one year old. Over the past year, we’ve built a world-class AI lab embracing both applied and fundamental research in machine learning.

From the start, we’ve organized around the ideas of transparency and openness that have made our field such an exciting place to be over the last few years. If big data and computing power have been the fuel for the spectacular growth of deep learning, then open-source publications and code have been the oxygen that turned the spark into a bonfire.

For that reason, we’re excited today to be launching the Element AI Lab Blog. Blog posts complement more “traditional” modes of publishing by giving us the space to discuss ideas informally, document work in process, and engage with richer forms of multimedia explanation. Many of us fondly recall taking our first steps in deep learning, aided by now-classic posts: Chris Olah’s beautiful articles on representations or Andrej Karpathy’s delightful dissection of RNNs. We hope that the Element AI Lab Blog can play the same role for future practitioners.

We’ll be embracing the whole gamut of content, from detailed discussions of research at Element AI and elsewhere, to practical tips for training and deploying models on the cloud. If you’re a research scientist at a prestigious university or a high schooler tinkering in your bedroom, we hope that you find something here to inform and inspire you.

Philippe Beaudoin, Senior Vice President of Research, Element AI

Archy de Berker, Applied Research Scientist, Element AI

Simon Hudson, Managing Editor, Element AI

To kick off our debut, we’ve included three blog posts to jump right into.

Bahador Khaleghi offers a thorough and thoughtful overview of the recent NIPS conference.

Jeffrey Rainy introduces our video style transfer project, Mur.ai, and follows up with a closer look at our process of using noise-resilience to stabilize live video.

Hello, World! was originally published in Element AI Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.

It’s a fairly safe wish to make, unless you tried to game the system by listing the exact opposite of what you wanted to achieve.

The sad truth is that resolutions are hard. No matter which strategy you rely on to reach your goal, it hinges on a single point of failure that we all share: a brain. A brain that will eventually choose a late morning in bed rather than early gym.

A brain that will convince you that missing the gym three days in a row means you’re a failure and you should just give up on your dream of being fit.

A web for your weak brain

Today’s web is optimized precisely for that brain. Under the pretense of “maximizing engagement” every web service exposes you to the links you’re most likely to click. It’s as if the web’s ultimate goal was to trigger every dopamine generator in your body. It promises Ten things people with strong brains do, number four will blow your mind!

Hint: number four is not following clickbait.

And if you ever train yourself not to click on such links, the web evolves on to the next drug. The current flavor seems to be badly cropped images that cut through a key element of the picture, inviting you to click to discover the missing funny, sexy or cute part.

Measuring happiness

Let’s go back a couple of years, when we all used Google to search for the information we wanted to see. How did Google become so good at placing the best result on top?

Like your brain, Google has some form of dopamine: it’s happy when you are… But since Google is a cold-hearted robot it needs to measure your happiness.

It does so by taking hints from your behavior.

What do you click on? How long do you stay on that page? Do you come back and edit your search query? Google takes every hint it gets, from its millions of users, and tries to design an experience that makes everyone happier.

And Google is not the only one doing this: Facebook, Snapchat and literally every web service rely on similar tricks.

That was all good and nice in the early days of the web, when pages were not actively designed to trigger each of your cravings. However, things have changed significantly since then and we’re beginning to see the limitations of these happiness-measuring robots.

Like genies from the legend, they are taking our wishes too literally. If I could have a small chat with Facebook these days here’s what I’d tell it: “Sure I clicked on this link, sure I spent an hour browsing pictures of the worst wedding cakes ever, but, looking back, I wish I had spent that time reading a book instead.”

The person you wish you were

Like Facebook, my best friend knows I love cake, but she also knows I want to cut down on sugar. Will she bake a chocolate cake for my birthday? Maybe, maybe not… She’ll make an enlightened decision based on how well I’ve been at keeping that resolution and how likely I am to regret eating cake the next morning.

She can do that because she doesn’t only know how I behave in front of a chocolate cake, she talks to me frequently enough to know how proud I am of what I do.

We all know our brain is weak, and in moments of strength we’re all able to reflect on this weakness. We can talk about what makes us proud as well as the behaviors we’d like to change. We do this on our own, we do this among good friends… But we never really do it with the tech giants that provide us with our beloved online services.

Focus on the user and all else will follow

Most of the criticisms we read about tech giants seems to start from the premise that their intentions are bad. That they care so much about money they will do everything they can to make you “engage” more and in the process they’ll reap the benefits of your clicks.

Having worked at Google during five years I can tell you this is not my experience. Google believes that if they focus on the user all else will follow, and I found this core value reflected all around me: in the engineers, the PMs, the VPs and all the way to the top. I’m pretty sure Facebook is driven by a similar philosophy.

In fact, I’m convinced that thinking any other way is a direct ticket to failure in the tech world and that’s one of the reason I’m so proud of being a part of it.

Less policing, more empowerment

If one starts with the idea that the tech giants intentions are bad, the solution for cutting down on “online sugar” will tend to take one of the following forms:

1. Ask that users police themselves by cutting down on online time altogether, or;
2. Request that the tech giants police themselves by dropping some links that are believed to be universally bad.

Both of these solutions could work in theory, but none are optimal. Some of the interactions I’ve had on Facebook have pushed me to become a better person, so I don’t really want to stop using it. On the other hand, after a long day of hard work I might be totally fine winding down by browsing a collection of bad wedding cakes, so I don’t want Facebook to get rid of these links altogether. The fact is that, like baking a chocolate cake for my birthday, it’s hard call. I want my online services to make an enlightened decision as to what should go on my stream or not.

If, on the other hand, you start with the idea that the tech giants really care about making their users happy, you can arrive to a totally different solution. One that empowers both the user and the service provider.

Something I call compassionate artificial intelligence.

Towards compassionate artificial intelligence

The problem with our cold-hearted robots is that they never have the opportunity to learn more about the person we wish we were, because we never talk to them outside of our goal-oriented interactions.

Therefore, the first step is to design user interfaces that let us have conversations that start with “I wish I had…” Such conversations cannot take place in the middle of an interaction. After I just wolfed down a huge piece of cake is rarely a good time to start thinking rationally about my behavior.

Instead, we need to design systems that let us have conversations in moments of strength, when we’re able to reflect rationally on the ways in which we’ve been weak.

A proposal for such a system would be a compassionate artificial intelligence. A simple conversational interface that you could access every night before going to bed. In this space, your personal compassionate assistant could summarize how you spent your day online (and maybe offline) and ask you how you feel about the different actions you’ve taken.

Some nights it could be a short conversation, focusing on the stuff it strongly suspects you regret or are happy with.

Some nights it could be a long-winded conversation, diving into your aspirations: the fact that you’d like to spend more time reading, or that you wish you’d start playing an instrument.

Such a system could be a natural extension of existing to-do applications, making them more intelligent and aspirational in the process.

Letting an online service know about your deeper aspirations might sound scary at first, but if you think about everything this service already knows about you, then it’s easy to see that such conversations would only allow you to tip the scale back towards a truer version of who you really are. In this way it becomes clear that a compassionate artificial intelligence is the missing link. It’s the tool that would make it possible to enlist the tech giants as trusted allies in the pursuit of our greater goals.

Tuning every dial

In this post I’ve focused on links we regret clicking, but these are only the tip of the iceberg. There is a myriad of mechanisms online services rely on to capture our attention: playing a ding sound, increasing a red notification counter, letting us know how many people liked our picture, notifying us that a friend is typing an answer, etc.

In fact, there are so many such techniques that timewellspent.io is keeping a collection of them as warnings to product designers.

Yet if we start from the perspective that online services are focusing on the user, and if we build a compassionate artificial intelligence that gives them more visibility into our aspirations, then all these dreaded mechanisms could suddenly become tools that help us reach our greater goals. The ding sound could play only when your long lost friend wants to have the conversation you’ve been waiting for. The notification counter could stay hidden until you decide you’re ready to wind down… And posts could be ranked not based on how likely you are to click them, but on how likely you are to be genuinely happy about that click.

All of this must be made possible not through countless controls and switches deep in a settings menu, but through a simple new way of looking at our online world: a web optimized not for who we are, but for the persons we wish to become.

Tap the ❤ if you like what you read & follow me on Facebook or Twitter!

A web for the person you wish you were was originally published in Startup Grind on Medium, where people are continuing the conversation by highlighting and responding to this story.

I’ve worked on a number of projects at Google, but the most famous one is probably the Chrome New Tab Page — the NTP. If you use Chrome you’ve seen it. It’s the page that pops in your face every time you launch your browser. At least until you replace it with an extension. Ahem.

What’s in there to “work on”, you ask? Good question. What should be a couple of lines of HTML turns out to be slightly more complex. Nowadays the NTP sports a fancy recommendation engine. Chrome is presenting you with eight thumbnails. Eight URLs it thinks you may want to navigate to.

It’s a feature that goes back to the first days of the browser, back in a time when no one dared call it a “recommendation engine”. Yet, even though things have changed under the hood, if you compare the current NTP to the original one they look fairly similar. In there lies the crux of the problem… A problem I bet many developers and designers will face in the near future.

Your AI in ball and chain

Turns out Google is pretty good at guessing what people like to do on the Internet. Are you watching Stranger Things on Netflix every night at 9PM? Did your coworker finally update that shared Google Doc you’re waiting on? Is there a centithread on reddit you absolutely have to chime on?

Beside being useful recommendations for the NTP, these examples all have something in common: they don’t work with the UI. Let’s see how:

• Thumbnails are hard to scan quickly. They work great if the list doesn’t change too often, but shuffling everything to add Stranger Things is the best way to confuse the user ;
• That Google Doc is only useful because your coworker edited it. If there is no way for the UI to convey that, there is no point adding it to the list ;
• If you start adding reddit posts, the list might quickly grow beyond 8 recommendations.

Every AI deserves a voice

The only thing I can predict about the AI in your product is that it will get brighter. Oh, you don’t have an AI? I bet you do! All these manual heuristics you coded are the seed of what may eventually become the brain of your software. If you don’t plan for it you’ll either miss opportunities or end up changing your UI more frequently than you should.

As it grows more and more clever, an AI becomes better and better at expressing itself. It can ask the right question at the right time or surface the relevant information when needed. This means your UI should be built with flexibility in mind. Forms with dozens of fields are a thing of the past, now it’s time to take the same route with the rest of your UI. Stop trying to coerce every bit of information into its specific little portion of the screen.

While your AI is growing a brain…

The thing with clever AIs is that they don’t exist. At first you’ll have a lot of “A” and very little “I”. If you’re not careful about it you may face the biblical flood of useless information that has plagued even the greatest — yes, I’m looking at you Google Now.

The key is to build a UI that is AI first but not AI only.

You should design a UI that works well even if your AI is only a set of hand crafted heuristics. It may sound impossible to do, but I’ve found the following guidelines help.

• Design good flexible units of information (FUIs).
  Think Google Cards, but tailored to your product. Make them really flexible: a given unit should be able to display as little as an icon and as much as a complex array of data.
• Arrange these FUIs in a layout that works.
  Units should be free to take the size they need and not more. Lists are great, grids are awful.
• Make FUIs expandable.
  Each unit is a guess your AI is making. Sometimes it’s confident and wants to display a lot of information. Sometimes it’s taking a chance and should wait for a click before adding the details.
• Think of your UI as a conversation with the user.
  Imagine you’re offering information, then asking a question, then refining the information in return. If you’re into conversational UIs they’re perfect for the job. Simply add your FUIs to the conversation stream. If chatbots are not your thing, design interactive units that allow the user to guide the conversation and summon new information.
• Sort, filter and fold aggressively.
  Remember: your first AI will be dumb. It will be able to make one or two good recommendations but great guesses will be rare. Sort the FUIs, keep only a limited number to avoid information overload, and collapse your wilder guesses.
• Have a universal fallback.
  A dumb AI means that sometimes the information users really want will not be shown. Adding a navigation page, a More Recommendations link or a search box will help them get where they want to go.

Let a thousand flowers bloom

Now that you have an AI first product, it’s time to reap the benefits. The biggest advantage is that you can now try things: start building FUIs, arrange the information differently, tweak your heuristics. The idea is to always be running experiments. Remember:

No data, no AI!

So collect data! Track everything the users do. Let them easily give feedback on the FUIs they see. A/B test and roll-out your new units gradually. Keep what works and tune what doesn’t.

And once you have enough data, unleash the AI gods! Your app is ready for it, your users are ready for it… and so are you.

Why you should think AI first was originally published in Startup Grind on Medium, where people are continuing the conversation by highlighting and responding to this story.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. NIPS 2012.

Link to full paper
Part of my Academic Origami series. How do I read this?

This is the seminal paper that introduces AlexNet, the deep convolutional neural network (CNN) that really kicked ass on the ImageNet contest. That contest has 1.2M images each belonging to one of 1000 classes. This paper launched the current craze over CNN. On the query “Convolutional Neural Network” Google Scholar returns ~1400 papers between 2000 and 2011 and ~5500 papers since 2012.

Convolutional neural networks are not that complex to understand, but there is a fair amount of specialized nomenclature that I will not cover here. If you want a quick introduction you can read Chapter 6 of Michael Nielsen’s Neural Networks and Deep Learning (although you should read the whole book!)

When it was published, this paper managed to train one of the largest CNN to date, which they did through a number of key improvements including:

• a very efficient GPU implementation and an architecture that allows training to happen on multiple GPUs,
• the demonstration that the now ubiquitous ReLU non-linearity significantly decreases training time.

And then, to reduce the very real danger of overfitting, they used a number of techniques including the dropout algorithm which had recently been introduced.

Remember that each neuron in a neural network computes the weighted sum of its inputs, applies an offset, and then runs the result through a non-linear function.

Historically, the most popular non-linear functions were the sigmoid or tanh. This paper makes a strong demonstration that the ReLU non-linearity significantly reduces training time on deep networks. The reason is likely that, for positive activations, ReLU’s derivative does not go towards 0. This is an important benefit since a derivative close to 0 means the gradient descent will converge very slowly.

Naturally, with negative activations the ReLU’s derivative is 0 which means it wont learn, but as long as some neurons have positive activations, then part of the network will be learning. It appears that it’s better to have only a subset of the neurons learning very efficiently than having most neurons learning very slowly.

This figure shows how much faster a network with a ReLU non-linearity (the solid line) learns compared to the same network with a tanh non-linearity (the dashed line). The magnitude of this effect naturally depends on the network and the task, but since this paper has been published, the advantage of ReLU has been confirmed in many other applications.

The proposed network is too large to fit on one GPU, so the authors propose to use the fact that modern architectures allow two GPUs to read and write from one another’s memory. Their final architecture is a column network, a term I’ve heard used a number of times since this paper. In a column network each GPU operates more-or-less independently except on some layers where they cross over.

Using two GPUs reduces their error rate by 1.2% - 1.7% compared to the single GPU network. Naturally, the single GPU network has roughly half as many convolution kernels as the two GPU implementation.

The goal of that slightly scary equation is to encourage different convolution kernels to learn different features. Remember that a kernel is a “matrix” applied on a square area of pixels. First, each entry in the matrix multiplies the pixel under it, then the neuron sums up everything, applies the offset, and passes it through the non-linearity. A problem occurs when two kernels have very similar “matrices”, which causes them to detect similar features, which in turn reduces the expressiveness of the network. This is what the equation here sets out to solve.

The equation first imposes an order to the kernels and looks at n adjacent kernels. The chosen order is totally arbitrary, if two kernels are adjacent it doesn’t mean they’re close together on the screen, it just means that the random order picked them to be neighbours. For each kernel, the equation inhibits an entry in the “matrix” if the adjacent kernels have a high value in that same entry.

In the above example, entry A of Kernel 3 will not be inhibited too much because the * entries of the adjacent kernels have a low value, whereas entry B will be more strongly inhibited because the † entries have a higher value.

Using this technique reduces the implementation’s error rate by 1.2% - 1.4%.

A quick note: I use quotes around “matrix” because a kernel is not really a matrix. For one, convolution is done through cell-wise multiplication which is not how standard matrix multiplication works. For second, the kernels are really tensors with 3 dimensions. For example, in this CNN, the kernels on the first layer have size 11×11×3 since the source image has three color channels.

Another improvement is the use of overlapping pooling. A pooling layer is typically applied after a convolutional layer. Its role is to downsize the image by gathering together a small window of pixels. In prior work, a given pixel was used only in one of the pooling neuron. In this paper they propose to overlap the pooling windows so that the same pixel can be used by two different pooling neurons. This reduces the error rate by 0.3% - 0.4%.

Fig.2 illustrates the full network architecture, with two clearly visible horizontal columns which cross over at layer 3. There are 5 convolutional layers and three fully connected layers. Here’s how to read that figure:

The input is a 224×224 image with 3 color channels. On the first layer, 96 kernels (48 per GPU) are applied to this image, each such kernel has size 11×11×3 and they are scanned over the image with a stride of 4 pixels on the X an Y directions. This results in 96 images of size 55×55 (48 such images per GPU) which are then run through a max pooling stage.

The figure is a bit misleading regarding the max pooling stage. It makes it look like this stage happens after the 5×5 convolution kernels. This is incorrect: max pooling happens before the 5×5 kernels are applied. Max pooling is applied in a sliding window of 3×3 pixels within the 55×55 images. This window is scanned over each of the 96 images with a stride of 2 pixels, leading to the overlapping pooling mentioned earlier. This downsizes the images by a factor of two, resulting in 96 images of 27×27.

Next, 256 convolution kernels (128 per GPU) of size 5×5×48 are applied on each of these 27×27 images, yielding 256 images (128 per GPU) of size 27×27. That stage clearly shows the impact of having two separate columns: each kernel is applied on only 48 of the 96 available images.

There are five stages of similar convolutions, typically happening independently on each GPU, except at stage 3 where a cross-over happens. After the convolution layers are three fully connected layers, the first two having 4096 neurons and the third one having 1000 neurons. A 1000-way softmax is applied just before the output.

Something that is not directly obvious from the figure is how many free parameters appear at each layer. Let’s look at a couple of examples. The first convolution layer has 11×11×3×96 ≈ 35k weights and 96 biases, the second one has 5×5×48×256 ≈ 31k weights. Between the last convolution layer and the first fully connected layer there are 6×6×256×4096 ≈ 37.7M independent weights (notice that the max pooling reduces the 13×13 images to 6×6, which is not shown in the figure). Finally, between the two fully connected we count 4096 × 4096 ≈ 16.8M weights.

A good way to make the most out of a dataset is often to augment it by distorting each sample in a way that ensures the label stays valid. The authors do this here by applying translations and horizontal reflections, increasing the number of training images by a factor of 2048 in the process. This data augmentation step is critical to limit overfitting. Another distortion they apply is to change the RGB color value of all the pixels every time the image is encountered in training. The proposed distortion captures the fact that natural images are invariant to changes in the intensity and color of the illumination. If you’re lost about how a PCA in RGB space works, just ask about it in a comment and I’ll help you figure it out.

The paragraph on the dropout algorithm is very clear, so I’ll let you read it. It’s interesting to know that, even with every mechanism described before, the network had a tendency to overfit which dropout fixes. Also interesting is the fact that dropout roughly doubles the required training time.

The paper has some more details on the parameters and mechanisms that were used during learning. There is nothing ground breaking here, but you should read the full paper if you want to implement the algorithm. One important detail is that the authors initialize the weights in each layer using the same Gaussian distribution. However, Nielsen convinced me that initialization was way too important to be done with uniform Gaussians, so I’m guessing the algorithm learning speed could be improved upon with more careful initialization. In case you’re curious, training on the full dataset took 6 days on two NVIDIA GTX 580 3GB GPUs.

ImageNet Classification with Deep Convolutional Neural Networks was originally published in Academic Origami on Medium, where people are continuing the conversation by highlighting and responding to this story.

Christian Szegedy, Alexander Toshev, and Dumitru Erhan. NIPS 2013.

Link to full paper
Part of my Academic Origami series. How do I read this?

The goal of this paper is to use deep neural networks to find these nice little “bounding boxes” you often see around objects or faces in computer-analyzed images.

The approach used here looks obsolete, the popular ones these days are Region-based Convolutional Network (R-CNN) and MultiBox. I still read this as it looked like one of the first DNN + localization paper. Also, the idea of using a mask to express the bounding box, as proposed here, is one you might come up with is you were faced with this problem.

In a nutshell: the output of the DNN is going to be a mask of “binary pixels” with 1 where the object is and 0 where it isn’t. The mask has a significantly lower resolution than the image.

In addition to the full object mask the DNN outputs four other masks covering the left, top, right and bottom of the object. This makes it possible to identify two adjacent objects. In this implementation, the original image is 225×225 and the mask is 24×24.

The result is a bit too coarse, so they refine it by cropping the image to a smaller window, then sliding this window across the image and merging the masks reported by each of these sliding windows. (The second line in Fig.2.)

There is another refinement process indicated in Fig.2 with the “refine” arrow: the algorithm “zooms in” on the detected bounding box, crops the image within it, and applies the DNN localizer again.

You can see some gray pixels in the mask on figure 2. It’s because a mask pixel value somewhere between 0 and 1 indicates that the real bounding box occupies only a fraction of the area covered by that mask pixel.

In this excerpt, the [14] is a seminal paper introducing the first deep Convolutional Neural Net (CNN) that kicked ass in the ImageNet competition. I’ve already origamized that paper, in case you want to learn more about it. The CNN in [14] outputs a vector over the classes of objects it wants to detect. Only one of these output should be close to 1 while the others should be close to 0, which is achieved using softmax.

In the current paper, the authors want to output a d×d image of mask pixels where each can have a value between 0 and 1. Therefore the softmax layer is replaced by a regression layer which, as far as I can tell, is just a layer of d×d regular neurons with no non-linearity applied.

That weird equation is the cost function used to train the DNN. The Diag(m)… part just means that every output where m = 1 will be multiplied by sqrt(1 + 𝜆) and every output where m = 0 will be multiplied by sqrt(𝜆). This way, mask pixels equal to 1 are weighted more heavily than those equal to 0. This makes sure the network doesn’t tend to make every mask pixel equal to 0 when the object is small. Note that the smaller the 𝜆, the more important the bias towards mask pixels equal to 1. I couldn’t find which value they used for 𝜆.

In section 5.1, the authors add a twist to their algo by using 5 masks instead of 1: a mask for the full bounding box, one for the top half of the bounding box, one for the bottom half, one for the left half and one for the right half. To understand which problem this solves imagine an image with two objects X and Y side-by-side versus an image with a single object Z. Their full object masks would look like this:

From these masks there is no way to disambiguate both cases. On the other hand, if you generated the left and right masks you’d get:

Where the optimal choice is two bounding boxes in the first case and only one in the second.

Equation (1) caused me a small headache the first time I read the paper. It simply indicates how to calculate the value of a mask pixel m at coordinates i, j given a bounding box bb and a type of mask h (full, top, bottom, left or right). The equation says that m should be equal to the percentage of the bounding box overlapping the mask pixel (when h = full) or the percentage of the half-bounding-box 0verlapping the mask pixel (when h = something else). A case of math-induced obfuscation if you ask me.

This paragraph basically says that the five d×d masks can be generated using a unique DNN with 5×d×d neurons on the output layer. The last sentence is strange to me: I was assuming they trained a single network with all the bounding boxes, no matter which class of object was in the image. Now it seems I might be wrong and they are training as many different networks as there are classes. This may mean 1000 different networks for today’s datasets!

This section explains how to find the bounding boxes (ie. the top-left and bottom-right pixels in the original image) from the five masks obtained at the previous stage.

For the full bounding box (h = full) they look for the bounding box bb that maximizes:

The 1/area term keeps the bounding-box as small as possible. The term in the summation is maximal when the value of the mask pixel m(i,j) equals the area of the bounding box covered by that mask pixel.

For the half-bounding-boxes (h = top, bottom, left or right), the equation to maximize is:

The first term in the sum ensures that the half-bounding-box precisely matches the mask pixels = 1, while subtracting the second term ensures that the the other-half-bounding-box (h-bar in the equation) matches the mask pixels = 0.

From this paragraph it appears that the full exhaustive search is too complex, so they limit it to 90 different bounding box shapes that are slid across the image with a stride of 5 pixels. Rather than defining the bounding box shapes with their width and height, they use the mean dimension (average of width and height) and the aspect ratio (width/height). This ensures that the bounding boxes are scaled relative to the image. They select 10 aspect ratios by analyzing the shape of the boxes in the training data.

Remember that a mask can encode multiple different bounding boxes whereas the exhaustive search looks at one bounding box at a time. To solve that problem, they try all the possible masks and keep the ones with strong enough scores. They then apply the classifier and discard the bounding boxes that do not work with the classifier. The non-maximum suppression algorithm is easy: first sort all bounding boxes by score then select them in order, dropping any bounding box that overlaps an existing one by more than 50%.

Again, the sentence “W.r.t to the class of the current detector” makes it sound like they need to train one neural network per class. This sounds overkill.

Section 5.3 explains how they apply the DNN on multiple crops of the image: the full image, and a number of sliding windows at 50% and 25% of the full image size. These sliding windows overlap by 20%.

For inference, they apply the DNN to all cropped images, at all scales (typically less than 40 cropped images). They obtain three masks, one for each of the three scales (100%, 50%, 25%), by merging the masks generated by all the sliding windows at that scale. Merging is done by taking the maximum of all the mask pixels at coordinate i, j. They then use the mechanism in section 5.2 to identify the top 5 bounding boxes per scale, obtaining 15 bounding boxes.

Another pass is made by “zooming in” on these 15 bounding boxes to obtain finer bounding boxes. First, the bounding box is enlarged by a factor of 1.2, probably to capture the fringes of the image, in case some information is present there. Then the image is cropped w.r.t. the enlarged box. Then DNN is run on that cropped image. I imagine they do not apply the multi-scale refinement at that point, just the basic DNN. Also, I’m guessing they only keep the the top bounding box.

I’m not sure I fully understand section 6, so this is my interpretation.

They train two things: first the classifier then the mask generator. The classifier is the CNN they use for pruning in section 5.2. I find it strange that they explain how they train it here, I had the impression it was the basic CNN of [14].

For training the mask generator, I believe they generate a training sets composed of pairs of (image crop, mask). 40% of these pairs are positive examples (the mask matches an object in the image crop) and 60% are negative examples (the mask doesn’t match anything).

The part about training the classifier I don’t get. Why are they talking about bounding boxes in this paragraph if it’s used to detect whether an image crop contains a given class of object?

Deep Neural Networks for Object Detection was originally published in Academic Origami on Medium, where people are continuing the conversation by highlighting and responding to this story.

Turning scientific publications into paper cranes

Origami is the art of transforming a piece of paper into a beautiful object, or, as often as not, into a crumbled ball in the basket. That’s what I intend to do with this new series of posts on scientific papers I read. Either share my understanding and help everyone marvel at an arcane piece of science, or blatantly expose my poor comprehension for everyone to see.

My hope in doing this is threefold:

1. Organize my own thoughts on a scientific paper,
2. Help others who might be struggling as much as I am,
3. Appeal to anyone who understands the subject and wants to help me.

If I’m honest there’s a fourth objective: encourage others to do the same so that I can read digested versions of papers in the future.

How does it work?

An origamized paper will be made from excerpts of the original paper (in the form of image captures) interspersed with my informal explanation, my interrogations and my opinions. The best way to read a post is top-to-bottom (no kidding!) then whenever you encounter an excerpt from the original paper, read it first before you read the text beside it.

Each paper I origamize will be published on Medium. Everyone is welcome to comment right on the published document using Medium’s highlight-and-comment tool. I’ll fold interesting recommendations into the main text, or at the very least answer them.

Principles

Any such grand undertaking needs a set of pretentious principles, here are mine:

• Focus on ideas, not people.
• Be casual and clear, not concise and formal.
• For previous work, see previous work.
• For the full paper, see the full paper.
• Opinions help.

Corollaries

I’ll refer to papers by what they talk about, not their authors. I know, I’m breaking an age-old academic tradition. Deal with it. :)

I’ll use the same casual language I use everyday in emails or on Facebook. This is how I like to write and, incidentally, that’s the kind of stuff I like to read. Yes, there will be smileys.

I will not cover previous work. I will aggressively de-duplicate information. I will aggressively cut parts of the papers I do not consider helpful, or which I prefer to rephrase. I will very rarely include results.

Last but not least, I will not shy away from voicing my opinion. The goal is not to discredit ideas, but rather to help readers build a better mental model of the field. Which idea had a critical impact on later papers? Which idea was a dead end? Which one sounds promising? What part of the paper seems impossible to understand? I’ll basically be offering answers to the questions I always have whenever I read a paper.

Keep in mind

1. I’m going to get things wrong.
2. I’m expressing an opinion on an idea, not on yourself. Also, see (1).
3. I’m not an expert. If I were, I wouldn’t be reading these papers, so (1).

Therefore please correct me. Always. Counter my opinion with your own. If you make a good case for it you’ll likely succeed in changing what I think. This will make me happy and the world will sing.

Conclusion

It’s an experiment. I’ll see how long it lasts. I’m reading a bunch of papers these days and I find that keeping a Medium post on the side isn’t too hard and brings me much benefits. I expect to keep doing it for a while. However if you think what I’m doing brings value don’t hesitate to let me know. A tap on the back works wonders for me. :)

And now, time to read some papers!

Academic Origami was originally published in Academic Origami on Medium, where people are continuing the conversation by highlighting and responding to this story.

Remarque: Ce sont des itinéraires d’été. Nos enfants avaient 8–10 ans.

En s’éloignant graduellement du centre-ville…

Granville island. Plus beau marché de Vancouver, situé sous un pont, c’est spécial. Touristique mais plein de locals quand même. Meilleure place pour stocker des trucs pour faire un pique-nique. Il y a du take-out aussi, ou de la bonne crème glacée. Pour les adultes: juste marcher, sentir les bonnes odeurs, profiter du beau temps et de l’air de la baie. C’est pas mal de marche mais il y a plein d’endroits pour s’arrêter. À voir pour les plus jeunes: il y a un souffleur de verre qui travaille souvent et qu’on peut voir d’une petite fenêtre; sinon il y a “le marché des enfants” où ils peuvent dépenser 5$ à jouer à des petites machines et gagner des billets pour avoir des toutous. Mes enfants adoraient ce truc. Petit parc sympa sur l’île avec une mini glissade d’eau. Note: Le stationnement peut être pas mal chiant, surtout le week-end.

False Creek. C’est la baie qui commence autour de Granville Island et qui s’étent jusqu’au Science World. Il y a une belle piste cyclable/voie piétonne qui en fait le tour. C’est un peu long à pied, en vélo c’est génial. De Granville tu peux marcher juste 15/20 minutes est c’est très beau. Il y a des maisons flottantes, ou de beaux petits condos sur le bord de l’eau. C’est une marche agréable. Il y a possibilité de prendre des bateaux taxis depuis Granville pour aller jusqu’au centre des sciences, je ne l’ai jamais fait. C’est peut-être un peu cher.

Science World + Chinatown. C’est près l’un de l’autre. On combinait souvent les deux dans une même journée, avec un repas Dim Sum. Notre place préférée c’est Floata’s à un jet de pierre du centre des sciences. Sans prétention, grande salle, très familial. Si vous aimez les Dim Sum je recommande. À côté de là il y a le Sun Yat Sen Chinese Garden. C’est petit mais l’entrée est gratuite et c’est spécial d’y marcher. (Il y a une section où on peut payer mais ça ne vaut pas la peine.) Le chinatown est spécial avec les étals en plein air qui vendent toute de sorte de trucs bizarres. Une petite marche de 5 minutes c’est assez.

Stanley Park + Aquarium. On ne peut pas vraiment rater l’aquarium, c’est vraiment le plus beau au Canada. Combinez ça avec une petite marche dans les sentiers autour. 30–40 minutes c’est bien. Si vous prévoyez beaucoup de temps dans le parc il y a deux belles plages: Third Beach assez isolée au pied d’une falaise (probablement la préférée des adultes); Second Beach moins plage mais un beau parc, une piscine incroyable et un gros espace gazonné. Le tour du parc en vélo est aussi très bien, même si je préfère False Creek personellement.

English Bay Beach + Denman Street. Dans le genre “plage en plein centre-ville” celle-ci est très belle. Il y a le Boathouse Restaurant juste en face. Bonne bouffe si vous aimez le poisson et les fruits de mer, kids friendly en masse sans être un St-Hubert, beaux couchers de soleil! Petite marche sur Denman Street, avec des petites boutiques et des restos pas pire aussi.

La Casa Gelato. Une place avec une centaine de sortes de crèmes glacées, dont plusieurs parfums bizarres. On peut goûter avant de choisir. Le quartier est pas génial. On y passait en auto avant d’aller sur Commercial Drive ou en sortant d’une visite au Science World. Les enfants adoraient.

Broadway Street à Kitsilano. Entre Vine St. et Blenheim St. Une de mes rues préférées à Vancouver. Pas achalandé comme le centre-ville. Pratiquement pas de touristes, mais plein de petits restos, cafés, etc. On aimait beaucoup les sushis de Broadway Sushis au coin de Stephen St. Simples, bons et vraiment pas chers! :) Sinon il y a des cupcakes, des bagels, etc. Recommandé pour une petite marche du midi, peut-être avant d’aller à Kitsilano/Jericho Beach.

Kitsilano Beach. Superbe plage, beau parc pour les enfants, marche intéressante le long de l’eau depuis Vanier Parc ou il y a un planétarium pas trop mal. Piscine extérieure géante. Je préfère personellement la plage suivante, mais celle-ci est très bien.

Jericho Beach et Jericho Park. Ma plage préférée. Le parc est très beau, l’endroit parfait pour un pique-nique ou un BBQ en plein air. Il y a un petit bush plein de lapins que les enfants adoraient. Sinon juste marcher sur la plage est très agréable. On peut facilement se rendre jusqu’à Spanish Banks, plus à l’ouest. Rendue là, la plage descend très lentement, en juillet c’est cool de s’avancer très loin en n’ayant dans l’eau qu’aux chevilles, ou alors jouer au frisbee sur le sable, etc. La vue sur les montagnes de l’autre côté est incroyable.

Pacific Spirit Regional Park. Un très beau parc, une alternative beaucoup moins touristique à Stanley Park. Je pense que les arbres sont plus jeunes mais je la trouve plus impressionnante. Ça fait “forêt de Endor” dans Star Wars: une forêt assez dégagée d’arbres gigantesques mais aux troncs plutôt étroits. Il y a des dizaines de sentiers. Mon accès préféré est par la 16ème avenue entre Blanca et UBC. Allez vers le sud à partir de là.

UBC Rose Garden + Museum of Anthropology. Peut-être mon musée préféré à Vancouver. Ça vaut la peine. Ajouter à ça une petite marche sur le campus de UBC qui est superbe. Le Rose Garden vaut la peine, peut-être une marche le long de Mail Mall. Il y a le Beaty Museum situé sur Main Mall. Le musée est OK sans plus, mais il y a un squelette de baleine très impressionnant suspendu à l’entrée qu’on peut voir sans payer.

Vers le sud…

Steveston + Fisherman Wharf. Situé à Richmond, un petit 30–40 minutes de voiture. On aimait bien aller y marcher. Beau boardwalk le long de l’eau. Possibilité d’acheter du poisson frais sur les bateaux de pêche. Il y a un très grand parc coin Moncton et No 1 Rd que les enfants adoraient. Possibilité de faire une belle marche de long de l’océan aussi sur W Dyke Trail.

De l’autre côté du Lions Gate Bridge, à moins de 45 minutes de route…

Water taxi + Lonsdale Quay Market. Une sortie qu’on aimait bien faire avec les enfants. Ce marché, dans North Vancouver est pas mal… Mais bon, pas certain que ce soit si intéressant si on n’a pas beaucoup de temps.

Grouse Mountain. Si vous n’allez pas à Squamish et que vous voulez voir la vue du haut d’une montagne, prenez la gondole de Grouse Mountain. Définitivement très touristique, mais pas un “attrape touriste” non plus. La vue est très bien, mais si vous avez le temps je préférerais soit Whistler soit Sea-to-Sky Gondola. (Voir plus bas.)

Lynn Canyon. Pas très loin de Grouse. Tout près de l’entrée il y a un pont suspendu, moins long que le Capilano Suspension Bridge mais beaucoup plus intéressant à mon avis. Le pont est sécuritaire mais ça a l’air d’un vrai (celui de Capilano a des côté en plexiglass qui font fake, c’est un attrape-touriste à mon avis :)). Presque pas de touristes à Lynn Canyon mais beaucoup de locals. On peut prendre une marche d’une trentaine de minute jusqu’à une jolie petite chute où les gens se baignent par beau temps.

West Vancouver. Marine Drive, dans le coin de 24th est joli. Petits restos sympas. C’est bien pour faire un arrêt dîné sur la route de Squamish.

Lighthouse Park. C’est une forêt sur le bord de l’océan. Beaucoup moins touristique que Stanley Park et les arbres y sont encore plus géants. Je pense que ce sont les plus vieux arbres dans le coin de Vancouver. Très belles marches à faire, vues superbes sur le phare et l’océan. Il y a plein de rochers que les enfants aimaient escalader. Rien de dangereux, mais les sentiers les plus intéressants sont peut-être un peu plus difficiles qu’à Stanley Park ou Pacific Spirit.

Whytecliff Park. Un parc pas sauvage du tout, mais très agréable et très bien situé. Il y a une petite île tout près du rivage accessible à marée basse, les enfants adoraient aller y grimper. Il y a moyen de faire du bateau dans le coin aussi. Endroit parfait pour un pique-nique.

Bowen Island. Si vous voulez prendre un traversier sans payer le total, vous pouvez prendre celui de Bowen Island pour un day trip, en laissant la voiture au terminus de Horseshoe Bay. Il y a de belles marches à faire sur l’île et un petit village sympa. Ce ne serait pas mon premier choix, cependant…

Plus loin, le long de la route vers Whistler…

La route sea-to-sky jusqu’à Squamish. Une des plus belles routes au Canada, avec la mer à gauche et les falaises immenses à droite. Les Shannon Falls, près de Squamish, sont très belles.

Sea-to-Sky Gondola. Nouvelle attraction, dans le coin de Squamish. Ma blonde et mes enfants l’ont fait la dernière fois qu’on y est allé et ils ont bien aimé. Il y a un pont suspendu en haut. Définitivement plus touristique, mais quand on visite Vancouver, ça vaut la peine de grimper en haut d’une montagne!

Whistler. Si vous poussez jusqu’à Whistler, allez-y par temps clair sinon ça ne vaut pas la peine. Assurez vous d’aller jusqu’au Peak Lookout. Le dernier chairlift est un peu épeurant, mais la vue de là-haut est tout simplement exceptionnelle. Le Peak-to-Peak Gondola ne vaut pas vraiment la peine. Il y a plein de bons restos dans le village de Whistler, dans le genre de Mont Tremblant.