Recognizing User Gestures on a Phone Screen
While working on my own indie game project, I ran into an interesting challenge: I needed to recognize user gestures on a smartphone screen. It was important not only to identify the shape of the gesture but also to accurately calculate how much the drawn gesture deviated from the ideal template. Initially, I tried to solve this problem using custom-built static neural networks. However, despite my efforts, I was unable to achieve stable and reliable results.
During my further research, I stumbled upon an algorithm called “$1 Unistroke Recognizer,” which ultimately solved my problem elegantly and effectively. In this article, I want to share my experience and discuss this algorithm in detail from a practical perspective.
A Brief Overview of the Algorithm
What makes the “$1 Unistroke Recognizer” so appealing? First and foremost, it stands out for its simplicity and high reliability. Unlike complex neural networks that require large datasets, this algorithm needs only a single example to learn to recognize a specific gesture. It’s important to understand that it analyzes the dynamics of the movement itself, rather than the final image. Therefore, shapes that look exactly the same but are drawn from different starting points or in different directions will be interpreted by the algorithm as completely different gestures.
The process of working with the algorithm is divided into two main stages. The first is the training phase, which is carried out only once. At this stage, we define reference gestures, and the algorithm stores their parameters in its database. The second stage is the recognition phase itself. During this phase, the algorithm processes user input “on the fly,” compares it with the stored data, and calculates with high accuracy exactly which gesture was drawn.
Training
Let’s take a look at the steps of the training phase for a single reference gesture.
Step 1
The reference gesture is fitted into a square of a certain size (I used a 251 x 251 square), distorting its proportions if necessary.
Step 2
The reference gesture is traced from the starting point to the end point, and its total length is calculated.
Step 3
In this step, the coordinates of the points along the path are calculated (usually 32 or 64 of them) so that the starting point coincides with the starting point of the gesture, the ending point with the ending point, and the remaining points are distributed along the gesture path as evenly as possible (this is precisely why we need to know the total length of the gesture path).
The figure below shows an example of how to construct keypoints.
Step 4
The coordinates of the resulting points, relative to one of the corners of the square (traditionally the top-left corner), are saved — training is complete.
Recognition
The recognition process also consists of several steps:
Step 1
The gesture drawn by the user is fitted into a square of the same size as the reference gestures (with distorted proportions if necessary).
Step 2
Key points are calculated for the gesture in exactly the same way as in the training phase (steps 3 and 4).
Step 3
In this step, the system iterates through the sets of stored reference gesture points.
Get Alexander Shevelev’s stories in your inbox
Join Medium for free to get updates from this writer.
For each reference gesture point, the geometric distance to its corresponding point in the gesture being recognized is calculated (remember that both gestures are inscribed in a square of the same size). These distances are added together.
Step 4
The smallest of the resulting total distances is selected — it will correspond to the recognized gesture.
Clarification
As mentioned above, the algorithm has only two phases: training and recognition. In principle, this is sufficient if the goal is to guarantee at least some result.
The problem is that with this approach, one of the reference patterns will always match the user’s gesture, regardless of what the user actually drew.
Therefore, we need to distinguish situations where a gesture is considered unrecognized.
To solve this problem, through trial and error and the search process, I arrived at the following formula:
F = (D / P) / (0.5 * SD)
where:
F (“fuzziness”) is the resulting value that indicates how accurately the gesture was recognized;
D is the total distance between points for the selected reference;
P is the number of points;
SD is the length of the diagonal of the square in which the gesture was inscribed.
If F > 0.2, the gesture is considered unrecognized. I obtained the coefficient of 0.2 through trial and error during my experiments.
Evaluation of how closely a recognized gesture matches the reference
For my indie project, I need not only to recognize gestures but also to quantitatively determine how closely they match the reference (certain game mechanics depend on this). For convenience in further use, this number should be in the range [0, 1], where 1 means that the gesture perfectly matches the reference.
At the same time, I need the value to be close to 1 rather than 0 in most cases — that is, I need to tolerate some user errors when drawing gestures.
As a result, I arrived at the following formula (I use Kotlin’s mathematical functions in it):
R = (1 — (F / 0.2).coerceIn(0, 1)).pow(U)
where:
R is the evaluation result in the range from 0 to 1;
F is the result from the previous formula;
0.2 is the cutoff threshold for fuzziness from the previous section;
U is the nonlinearity coefficient. In my case, its value is 0.25. The smaller this value is, the more carelessly the user can draw a gesture.
If you are not familiar with the Kotlin language, then: the coerceIn() function converts the number it is called on to a specified range; the pow() function raises the number it is called on to the power specified in brackets.
Automating Training
As mentioned above, for the algorithm to work, you need a set of reference gestures during the training phase. It would be convenient if these gestures could be specified in a visual way. And it would be good to automate the training process itself.
I solved the first part of the problem by representing the gestures as images like these:
In these images, the gestures are already inscribed within a square of the appropriate size. The red dots specify the traversal order — the images are traversed in descending order of the R value in the dot’s color.
As for automation, I achieved it simply by using AI to generate a Python script that processes these images, identifies key points, and saves them to files (more on this in the next section).
Source Code
To debug this approach, I used the code available in my repository here.
It consists of two folders: Parsing and Recognition.
The Parsing folder contains reference images of the gestures I need and a generic Python script that handles the training phase — it analyzes these images, calculates keypoints, and writes them to files.
The Recognition directory contains a simple Android app that, based on the data from the training phase, recognizes the user’s gestures on the phone screen using the approaches I described above.
You are free to use this code in your own projects.
Happy coding!
