The key idea is to load the image with the flag cv2.IMREAD_UNCHANGED that tells OpenCV to keep the alpha (transparency) channel. Then we save separately the alpha channel as a binary mask for latter use.

Notice that when the original image has no alpha channel the mask will be simply returned as None.

Now it is possible to correctly extract SURF features from the masked image

A few weeks ago I was in a supermarket with my girlfriend, we were invited to a dinner and we wanted to bring wine. Neither of us is a wine expert but she had a fantastic ace in the hole, she pulled out her phone, took a picture of the wine label and immediately got a description and a review of that bottle. I was impressed by the power of that app (it is called ViVino) and she challenged me to build something similar.

I accepted the challenge but added a few personalizations to it:

• Since I prefer beer to wine I wanted to make a beer recognition app
• I wanted to be better than ViVino and be able to recognize even multiple bottles from a single photo.

This is how I came up with BiBirra (birra is the italian word for beer)

Scraping

The first thing we need to build a beer recognizer is a database of beer images. To my knowledge no such database exists so there is only one thing we can do:

Luckily the web is full of online shops selling beers and I’ve already developed a small library for web scraping called Raschietto. This library allowed me to write the scrapers really fast and in less than a day I was able to collect ~2'300 beer bottles pictures together with their basic information (name, type, color, alcohol by volume, bitterness…) from the web.

The information about each beer had some missing data here and there (mainly because some shops don’t contains much information about beers), cleaning up this data to get a polished result like in ViVino is really time consuming and requires a lot of hand work. My goal was to create a working prototype and the images where good to go, so I decided to proceed to the next phase.

Bottle Detection

Before doing the actual beer recognition we first need to detect every bottle in the image. Since the shape of a bottle is pretty much unique this could be done quite easily. One way to do this is to train a binary (two class: beer or not beer) classifier and then apply a sliding window approach to detect each bottle. Concretely this means applying the classifier at different locations and scales all over the input image and then apply Non Maximum Suppression to remove overlapping bounding boxes.

The classifier can be built using the classical HOG+SVM combination, which is very good in detecting objects which don’t exhibit too many changes in shape (for example humans). We just need to extract an Histogram of Oriented Gradients (HOG) from each training image

and then train a linear Support Vector Machine (SVM) on the transformed data.

This time honored approach is quite powerful but training such a classifier is time consuming, especially when we consider that we can get a modern deep learning-based off-the-shelf model for free and in almost no time.

Single Shot Detectors

For the detection I decided to use a particular type of CNN (Convolutional Neural Network) called Single Shot Detector. This neural networks directly output a set of bounding boxes and classes achieving detection without using a sliding window.

A big advantage of this architecture is that each pixel of the input image is fed only once to the model (compared to many times when using a sliding window) this means that we don’t waste resources recomputing local features many times. This observation motivates the name of a famous detection model: YOLO (You Only Look Once). Many similar models exists: SSD, R-CNN, Faster R-CNN.

I’ve decided to use the model presented here which is a combination of MobileNet architecture and the Single Shot Detector framework which can detect: airplanes, bicycles, birds, boats, bottles, buses, cars, cats, chairs, cows, dining tables, dogs, horses, motorbikes, people, potted plants, sheep, sofas, trains, and TV monitors. This model by using the MobileNet architecture is lightweight but still has good performances.

I just need the bottle class so this model is probably an overkill but it works quite well (and the recognition algorithm is robust to detection errors), is fast and ready to use so I’ve decided to adopt it.

Beer Recognition

Here comes the key part of the algorithm, given the image of a bottle we need to come up with a beer label. If we view this problem as a classification one we have 2’300 classes with 1 or 2 examples per class, an impossible problem. A better approach is to view it as a Content Based Image Retrieval (CBIR) problem: given a picture of a beer we need to find in our database the most similar image. The only thing we need to do is define (or learn) a similarity measure and use it to compare the query image to all the images in our database and chose the most similar one. The process of retrieval can be divided in two steps:

1. Embed: from each image we need to extract a succinct representation of its content. We then use this representation to compute the similarity between two images. One example of embedding might be computing the color histogram or using the latent representation learnt by an Autoencoder.
2. Index: given the embedding of the query image we will need an efficient way of retrieving the most similar images from the database. Indexing algorithms comes to our rescue, allowing us to do fast nearest-neighbors queries.

SIFT/SURF Features

Each beer label is unique and fixed (except for changes in perspective and light) therefore a good representation can be obtained by using SIFT (or SURF) descriptors. What these descriptors does is finding a set of keypoints on the image (together with their scale and orientation) and then representing each keypoint with a float vector (for example SIFT produces a vector of length 128). These descriptors are engineered to be robust to changes in scale, rotation and lighting.

With local descriptors like SIFT or SURF one can use many approaches:

• Use a bag of visual words approach representing each image with an histogram of descriptor frequencies and then compaing the histograms to find the image in the database most similar to the query image.
• Use the local descriptors together with RANSAC to try to align the query image to every image in the DB and then chose the best matching one. As proposed in the paper “Mobile Wine Label Recognition”.
• Given a query image, for each descriptor find the most similar descriptors from the DB and give some points to the corresponding images. After using all the descriptors chose the image with the highest scores as a match.

I’ve found the latter approach (which I think I’ve invented) to work better in this case, so I’ve decided to adopt it. Notice that this approach doesn’t require any alignment phase so it is faster than the method proposed in the paper and, by querying over all the images at the same time, allow the use of an index to speed up queries.

To further make queries faster I’ve compressed the descriptors down to 16 dimensions using PCA. This compression does not affect accuracy (it removes a lot of unimportant features) but reduces the space and time requirements quite a lot.

Approximate Nearest Neighbors with Annoy

To get fast nearest neighbors queries I decided to use Annoy (developed by Spotify)

Annoy (Approximate Nearest Neighbors Oh Yeah) is a C++ library with Python bindings to search for points in space that are close to a given query point. It also creates large read-only file-based data structures that are mmapped into memory so that many processes may share the same data.

Under the hood Annoy creates a forest of binary trees each built by splitting data with random hyperplanes. Each query will return a set of approximate nearest neighbors. The results produced by Annoy are only approximate, there is no guarantee of getting exactly the closest points, but this is the price to pay to mitigate the curse of dimensionality and get good performances.

Results

I’ve gathered a very small test set of 45 beer images on which I achieved 100% accuracy. Such a small test set is not statistically significant but this result is certainly a sign of good performances.

Let’s see the algorithm in action:

You can find more information about me, all my projects and all my posts on my website https://matteo.ronchetti.xyz

Other articles I wrote about Computer Vision:

Creating Pointillist Paintings with Python and OpenCV

BiBirra: Beer Label Recognition was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

• Using CSS selectors. I have worked as a fullstack developer so I prefer using CSS selectors than XPath.
• Creating a working scraper in almost no time. Most of the times I don’t need something very complex or advanced, I just need to make a simple scraper very quickly.

This motivated me to create a simple scraping library called Raschietto (the Italian word for scraper). Here are its main features:

• CSS Selectors
• Single line getting and parsing of an HTML page
• Simple and extensible methods for matching elements in the page
• Prebuilt matchers for common tasks (for example extracting links and images)

The library is currently in beta (but is sufficiently stable) and supports only Python3 (adding Python2 support is quite straightforward).

I’m sharing this library because I think it can speed up the process of building simple scrapers and because its small codebase (<170 lines of code) can be useful for learning purpose.

You can install it with:

pip install raschietto

Scraping arXiv

As an example to learn how to use raschietto and to showcase its capabilities I will guide you trough the scraping of an arXiv webpage. Please note that arXiv has a very nice RSS feed so its data can be easily accessed without the need for scraping, but we will still scrape it for educational purpose.

We will focus on the page that contains the latest machine learning papers.

As a starting point we begin with just getting the titles of the latest papers. Since raschietto uses CSS selectors this task is quite easy. Using Chrome we can copy the CSS selector that will select a specific element using the dev tools. Just place the mouse on the desired element (in the DOM view)> right click > Copy > Copy selector

We get the following selector:

#dlpage > dl > dd:nth-child(2) > div > div.list-title.mathjax

This selector is specific to that title element so it will not select other title elements. By looking at the HTML code of the page we can modify it to get a simpler selector that will select all the desired elements:

.meta .list-title

If we are unsure about the correctness of our selector we can test it using the console (in the dev tools)

The code for getting the titles of the papers with raschietto is straightforward and self explanatory

We are using two raschietto components:

• Raschietto is a class that contains all the main functionalities of the library. In this example we are using it to get and parse an HTML page
• Matcher is a class that encapsulates the matching logic. As a default it return the inner text of the matched element(s), we will later see how to change this behavior.

We now get as a result a list of titles that all start with “Title: ” to remove this first 7 characters we can use the mapping capability of the Matcher class

The only change we made is adding the mapping argument to our matcher. This is a function that takes 2 arguments: the matched element, and the containing page. We use a Raschietto function to map the matched element to its inner text and then use python slicing to remove the first 7 characters, obtaining the cleaned up list of titles.

We now wish to do something more advanced: we want to match both the title and list of authors for each paper. To achieve this we need to:

1. Match all the “.meta” elements
2. For each matched element get the contained paper title
3. For each matched element get the contained list of authors

The mapping capability of the Matcher class make this task really simple, we just need to match all “.meta” elements and map them to a dict containing the title and the list of authors.

With just a few lines of code we have created a nice scraper!

Of course this is not enough for us: we also want to get the link to the paper. This may seem problematic because of the structure of the arXiv page.

The title and the link are contained by different elements and it is not useful to use their first common parent because it contains all the papers elements. To get the link we have to:

1. Match the “.meta” element
2. Select its parent (the “dd” element)
3. Select the element right before the “dd” element (so we get the “dt” element)
4. Match the “a” element that links to “https://arxiv.org/abs/<paper id>”
5. Extract the href from the matched link

We already went through step 1. To achieve step 2 and 3 we can leverage the fact that the “el” parameter passed to the mapping function is an instance of lxml HtmlElement so we can use lxml methods to get the “dt” elemen by selecting the previous element of the parent of el.

dt = el.getparent().getprevious()

To complete step 4 and 5 we can use a prebuilt matcher that matches links , extracts the href from each of them and make each url absolute (for example it maps “/abs/1802.08246” to the absolute url “https://arxiv.org/abs/1802.08246”). Matching links is quite common when scraping so I’ve included this matcher in the raschietto library. This matcher has also a parameter startswith that, when specidied, will tell the matcher to keep only the results that begins with the given string. Using this matcher the code for our task is very short:

link_matcher = Matcher.link(".list-identifier > a", startswith="https://arxiv.org/abs/")

With this changes we get the final version of our arXiv scraper

Notice that we passed the page argument to the link_matcher to let it know the source url of the element (otherwise is impossible to make the url absolute)

Conclusions

With just 15 lines of elegant code (empty lines don’t count) we developed a working scraper capable of extracting information from a webpage. We were easily able to group the matched elements, overcome the problem of “no useful common parent” and extract the needed information from each element without writing a lot of code.

I truly hope you like this small library. If you have some questions about it feel free to ask them in the comments. If you want to collaborate to development of the library don’t hesitate to contact me :)

You can find more information about me, all my projects and all my posts on my website https://matteo.ronchetti.xyz

Raschietto: a simple library for web scraping was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of my favorite painting techniques is pointillism, a technique of painting in which small distinct dots of color are applied in patterns to form an image.

This technique relies on the ability of the eye and mind of the viewer to blend the color spots into a fuller range of tones. The juxtaposition of different colors make them more vivid, giving the whole image more vibrancy.

Pointillism is a gorgeous technique, but in order to make these types of paintings, you need to take a methodical and long approach. Not to mention, you need to be a skilled painter!

As a computer vision developer, I know a lot about processing images — but let’s face it, I’m terrible at drawing.

During my trip to Paris I was inspired to try to emulate pointillists using Python and OpenCV. In this post I will guide you through the approach I developed and the code I’ve written to produce images like the one at the top of this blog post.

JUST LOOKING FOR THE SOURCE CODE? YOU CAN FIND IT HERE

Simulating the Painting Process

The painting process simulation is done in three steps:

1. Color palette creation
2. Stroke size and length computation
3. Actual painting

As an example I’ve chosen a beautiful picture of Paris from https://unsplash.com a website where you can get free pictures of amazing quality. I’ve preprocessed the image with The Gimp to reduce the green shades on the gargoyle and to upscale the Tour Eiffel to make it more eye catching. In general doing a little bit of preprocessing, especially increasing the general saturation, helps getting more pleasing results.

Color Palette Creation

First thing we need to decide which colors will be in the palette of our “virtual artist”. We need to choose a set of colors that is coherent with the colors of the image but yet vivid and distinct.

We start by running k-means to select n=20 dominant colors from the image. The “adversarial” nature of k-means produces distinct colors that concentrates around the most frequent colors in the image.

To add vibrancy we augment the palette by adding some variations of the colors chosen by kmeans. I’ve chosen to add a more saturated version of the base palette, and two small variations in the colors (± 20 hue) with a little bit of added saturation.

The first row of the following image represents the palette computed by kmeans, the subsequent rows contains the color variations included in the final palette.

Stroke direction and length computation

To give our final result a more dynamic look we can use small distinct strokes of color instead of circular dots. This is actually a technique used in a lot of pointillist paintings! We want the direction of the brush strokes to be coherent with the content of the image and to use longer strokes to depict edges. For this reason we use the image gradient to decide the length and direction of each paint stroke.

To compute the direction and length of the brush strokes we take the Scharr image derivative on the x and y axis thus defining a vector field over the image. We then smooth this field using Gaussian blur to give a more coherent look to the final “painting”.

The Painting Process

We paint the final results just like a real artist, one stroke at a time! For each stroke we have to decide:

1. The position
2. The orientation and length of each stroke
3. Which color of the palette to use

It may be tempting to chose a random position but this approach has a major drawback: we don’t know when we have painted enough strokes to fill the entire image. So we create a list of strokes positions by sampling uniformly over the image, we add a little bit of random noise to each position and we shuffle the list to give the strokes a random order.

The direction and length are easy to compute once we have defined a vector field over the image using the image gradient.

We rotate the direction of the gradient by 90 degrees because we want the strokes to be parallel to the image edges (the image gradient is perpendicular to the edges). The length is computed depending on the magnitude (i.e. strength) of the image gradient. Here stroke_scale is a parameter that specify the global scale for the strokes, the square root is used to make the strokes more uniform. We don’t want to have too much variance in the length of the strokes.

The choice of the stroke color is probably the hardest part. We want it to be random to get that beautiful juxtaposition of colors, yet we want to be consistent with the source photo. The idea is to chose a color randomly from the palette, giving the colors that are more similar to the underling pixel of the photo a greater probability of being chosen. The probabilities are determined using a softmax function of the similarities between colors.

The Code

You can find the source code here. It corresponds to the algorithm I’ve described here, with some performance improvements. The code is commented and should be quite easy to understand. If you have any questions feel free to ask in the comments :)

Results

Some results with other photos:

I hope you enjoyed this article! If you have any questions feel free to ask in the comments. If you produce some great paintings please post them in the comments, I’m really looking forward to see what you can do with this program!

You can find more information about me on my website https://matteo.ronchetti.xyz

Other articles I wrote about Computer Vision:

BiBirra: Beer Label Recognition

Creating Pointillist Paintings with Python and OpenCV was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.