Assistive Robot: Recognize and Engage with People in Need
Introduction
In commercial places such as stores, airports, etc., we are often faced with situations when there are not enough attendants to fulfill customers’ needs. Robots can fill the gap to assist and engage customers autonomously. However, it is very ineffective for robots to attend to every customer in the space blindly. Instead, robots need to recognize customers that are in need and approach them to assist.
To create an assistive robot that can recognize people in need, we proposed an end-to-end video classification framework to classify 4 human actions (looking around, waving hands, phone calls, and playing with the phone) on raw RGB videos. Our approach used OpenPose for key-point extraction and classified the extracted key-points using the Decision Tree model and the LSTM networks. For model training and testing, we generated a video dataset consisting of both self-made videos and videos from public datasets. The framework achieved a test accuracy of 87% classifying 4 action classes, and test accuracy of 95% classifying “in need/ not in need” actions.
The authors were graduate students (Kenny Jin, Wenjun Chen, Xiaoyu Su, Jiongxin Ye, Yuzheng Jia) at Columbia University’s Data Science Institute (DSI), and this work was done in collaboration with mentors from J.P. Morgan AI Research.
Datasets
Action classes
The concepts of “in need” and “not in need” are relatively vague, and there could be various human actions that can be considered “in need” or not. Therefore, in addition to the 2 categories that we defined above, we further introduce some specific action classes for each of the 2 categories in the table below.
For simplicity, we assume that there is no more than one person in the scene and only use video data that contain only one person.
Self-made videos and open-source datasets
We collected video data for the 4 actions above by ourselves. We mostly recorded videos in the “Look Around” class since open-source datasets do not contain videos of this class. A recorded video would be automatically broken up into segments of 8–10 seconds. Thus, a single round of recording can generate multiple data points.
To enrich our training dataset, we decided to use open-source datasets combined with our self-made data. We found 2 public datasets we can utilize: KTH and NTU. You can find information about these 2 datasets on the KTH dataset website and the NTU dataset website.
Methods
Key-point extraction and data preprocessing
We established the structure of our data processing pipeline. Here are the steps to process every single video:
- Use OpenCV to detect bounding boxes of the person and return a folder or folders of images (depending on whether the video contains multiple identifiable people).
- Use OpenPose to process an image folder and return a folder of JSON files specifying the key points of the detected human skeleton. (The data of key points are stored as a list and contains the x position, the y position, and the prediction confidence for each keypoint. The order of the key-points is fixed and can be found on the OpenPose official website).
- Given the folder of JSON files, we transform and normalize the data so that the range of x positions and y positions are both [-0.5, 0.5], and output it in a structured format. Each row includes the label of the video, the frame number, and normalized key-point coordinates.
We embedded these steps in an end-to-end script that takes a folder of videos and outputs a CSV file with all the key-point data. The processing of these videos is easily parallelizable but is unnecessary given the limited number of videos we have at our disposal. The code of this pipeline can be found in our Project Github Repository.
Get Wenjun Chen’s stories in your inbox
Join Medium for free to get updates from this writer.
Different from other datasets that provide raw videos, the NTU dataset contains key-point data directly and we do not need to process it using OpenPose. Although the key points from OpenPose and NTU are slightly different, we are still able to use the NTU dataset after mapping the NTU key points to the OpenPose key points. We also normalized the NTU dataset so that its key-point positions have a similar distribution to those of OpenPose.
After all the data processing, we got a CSV file containing key-point data of 3736 videos in total, including 1024 self-made videos, 2260 NTU videos, and 452 KTH videos. We split the data into 80% train set, 10% development set, and 10% test set. The dataset remains the same for all downstream tasks.
Decision Tree Model
The decision tree is a supervised machine learning model that can naturally handle multi-class classification problems and is a good choice for small datasets. Every node of the tree will split the training data based on the values of available features to produce a good generalization, and the leaf nodes can refer to any of the K classes concerned. Moreover, a decision tree model is very intuitive and easy to explain to technical teams and stakeholders.
We designed 62 features for the decision tree, with Gini impurity as the criteria for decision tree splits. We selected the best hyper-parameters for the decision tree using cross validation. According to the experiment results, we set the maximum depth of the tree for our Decision Tree model as 9 and the minimum number of samples required to be at a leaf node as 1.
LSTM Networks
Recurrent Neural Networks (RNNs) can deal with sequential data with variable lengths whose applications are helpful in action classification problems. Unlike the Decision Tree model, we do not use hand-crafted features for LSTM. Instead, we utilize the time-series data of key-point positions directly.
We mapped the data into the matrix form that the LSTM model takes as an input. The sequence length is set to be 100 frames: any video with more than 100 frames is trimmed, and any video with fewer than 100 frames is padded with zeros. Therefore, the data point is a matrix tensor of size (100, 50).
Considering the limited size of our dataset, we built a shallow LSTM network consisting of 2 layers. The number of features in the hidden state was set as 100, while other parameters went default. The model is optimized over the cross-entropy loss with the Adam optimizer, where the initial learning rate is set to be 0.005 and decays by 0.99 every epoch. The model is trained for 100 epochs.
Results
Decision Tree Model
The test accuracy for the 4 action classes classification is 87% and the detailed classification results can be found in the table below. In the confusion matrix, we can see that 29.55% of the “Play with Phone” data were misclassified by the “Phone Call” action. The possible reason is that, as shown in the visualization of 4 action classes, some “Play with Phone” actions are similar to the “Phone Call” action, especially when people are playing with phones with only one hand. It can be one of the directions to improve our model performance in the future work.
As our goal is to recognize customers that are “in need” or “not in need”, we converted “Look Around” and “Wave Hand” tags to “in need” tags, and also converted “Phone Call” and “Play with Phone” tags to “not in need” tag in our final prediction. We were able to achieve 95% test accuracy for our final prediction results (i.e. “in need/ not in need”).
LSTM Networks
The test accuracy for the 4 action classes classification and the “in need/ not in need” classification are 88% and 94%, respectively. The detailed evaluation results and the corresponding confusion matrix are shown below. Observe that the LSTM networks and the Decision Tree model not only had similar accuracy rates but also made similar errors, conflating “Phone Call” with “Play with Phone”. This suggests that some of the skeleton movements in these two classes are indeed similar to each other and key points with finer granularity might be needed for better performance.
Overall, this model achieves satisfactory accuracy with a relatively small dataset, suggesting that sequential models indeed look promising for the task at hand. When more labeled data are available, we believe that these neural networks would come out on top.
Conclusion
Starting from scratch, we performed a preliminary exploration of recognizing and classifying human actions in commercial places. We generated a balanced dataset by videos recorded by ourselves and open-source videos and then retained the skeletal joint features from the RGB videos using OpenCV and OpenPose. We realize that collecting and organizing data is a crucial part of any real-world data science project.
In the Decision Tree model, we designed 62 features that are critical to train reliable decision rules used for classification. Then we attempted to cut out the feature engineering process and fed the data directly to a shallow LSTM to achieve comparable evaluation performance. Both the Decision Tree model and Long Short Term Memory (LSTM) networks delivered satisfactory classification performance: the test accuracies for “in need/ not in need” classification are 95% and 94%, respectively.
Thank you for taking time to read our post! We hope our work can provide some new ideas for you. Happy predicting!
