Intro of Posture Detection, & thinking more on OpenPose II
Posture Detection Review
The human body posture analysis attracts more and more attentions, as posture is one of the very important biological characteristics of personal body. Posture detection therefore become one of popular research topics and promising field as it can be applied to various scenarios, such as gait analysis, video surveillance, augmented reality, human-computer interaction, finance, mobile payment, entertainment and games, sports science, etc.
Posture detection allows the computer to know what a person is doing and recognize who is this person. Especially under the monitoring situation when personal face part is too small to identify from far away cameras, it is proved to be an effective solution and meanwhile personal privacy intensive information is protected. It can also be used as a significant auxiliary verification method in the target identification. Considering health monitor and health care are becoming attractive and grow up to the billion dollars market when aging problems deteriorate in majority of developed countries, automatic posture detection and intelligent decision making are highly valued and beyond academic research interest.
Human Keypoints Detection, also known as human body pose estimation, is a famous approach to realize human gesture detection. For normal circumstances, the human body key point detection can be subdivided into single or multi-person key point detection, 2D or 3D key point detection. We also usual adopt key point tracking after the key point detection is completed, which is known as the human body posture tracking.
COCO keypoints challenge is one of the authoritative public competitions. The COCO data set represents human body keypoints as 17 joints, that is nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, left and right ankles. The assignment is to detect the human body and the corresponding key point position from the input image.
For 2D multi-person key point detection, two approaches are widely used, that is top-down and bottom-up. Top-down approach is to implement target detection to find one person before implement single-person key point detection for each detected person. Bottom-up approach is to detect all the key points and group them person by person. Generally top-down approach is more accurate while bottom-up approach is faster. [1]
OpenPose techniques [2], one of bottom-up approaches, is receiving more and more attentions, based on a) OpenPose achieves better trade-off and gain high accuracy and fast response [3–8]; b) OpenPose provides open-source library and hence can be easily used [9–15]; c) OpenPose is widely applied and researched [16–21].
After OpenPose
After 2020, Cao et al. continued to enhance OpenPose [22] in order to improve detection accuracy and reduce computation complexity, including optimized architecture design, convolutional kernels optimization, enhanced PAFs, combining detection.
a) The newly proposed architecture is shown as below. Compared with the previous one, the two branches to calculate confidence maps and to calculate PAFs respectively are changed to cascade structure rather than parallel structure. What’s more, based on stimulation experiments, PAFs are more effective than confidence maps so that we can increase PAFs branches to depth, and setting PAFs as prior performs better. The authors giving interpretation is that output of PAFs contain the body part locations information while output of confidence maps does not contain the body part associations information. In addition, the new architecture can reduce the computation workload to half.
b) The previous 7x7 convolutional kernels are replaced by 3 consecutive 3x3 kernels23, so that the receptive field is maintained but the computation workload is decreased nearly to half.
c) The authors further enhance PAFs by introducing redundant PAF connections, such as ears to shoulders and wrists to shoulders. The technique improves accuracy for crowded images. Besides, the new algorithm cuts off possible pairwise connections whose 2 body parts have been assigned b different people, in order to further enhance performance of overlapping people situations.
d) The authors annotated a small dataset of foot keypoints and train model through the previous dataset and the small dataset to detect body keypoints and foot keypoints together. The experiment result demonstrates accuracy will increase for both body and foot detection (by a large margin), while inference speed is preserved.
Through these optimization techniques, the model’s accuracy and inference speed are further enhanced. Interesting, the new model performs much better when using GPU but much worse when using CPU because higher memory time consumption. Moreover, the model proves high capability of transferring to another keypoints detection tasks, such as vehicle pose estimation.
The new model performs not as good as expectation when dealing with rare pose, overlapping objects, shared parts, occluded parts, invisible body part (this demonstrates the new model put high weight on body part detection because of imbalance of training dataset).
OpenPose Various Applications
OpenPose algorithm has not only been enhanced, but is also widely used into various applications to prove its decent quality and robust for diverse backgrounds, here including 3D posture tracking and falling detection.
D’Antonio et al. in 2020 proposed a method to detect lower limb joint angular kinematics based on OpenPose [24]. The basic idea is to use two pairwise-calibrated cameras and process by OpenPose and calculate by triangulation to output lower limb joints angles. The detected subject is walking while wearing IMUs on lower limb keypoints as benchmark. Interesting, results show OpenPose method is somehow inaccurate and produce angular error values up to 9.9°, while nevertheless OpenPose is qualified for the estimation of the range of motion of lower joints.
Nakano et al. in 2020 analyzed the 3D motion tracking by multiple cameras processed by OpenPose [25]. Benchmark is optical marker-based motion capture with 48 reflective markers on the body and 16 cameras used, compared with OpenPose markerless motion capture with 5 cameras and direct linear transformation method to output 3D coordinates. Experimental results show 47% time were <20 mm mean absolute errors, 80% were <30 mm, 10% were >40 mm. The errors originate from OpenPose false recognition of an object as a human body part or occluded human parts, and moreover taking no account of temporal information (keypoints moving continuity).
As OpenPose may cause false recognition out of subject self-occlusion and confusion about left and right side, in 2020 Slembrouck et al. proposed a method to utilize the 3D reconstruction from multiple cameras by using least squares triangulation [26]. The method can achieve real time tracking at 20 to 25 fps and prediction errors standard deviations between 6.6 and 21.3mm.
Falling detection and prevention attracts more than 1000 research papers focus, which can be classified into 4 directions including inertial sensors approaches, context based sensors approaches, RF based sensors approaches, fusion based sensors approaches. Context based computer vision is one of the most significant, convenient and mature approaches [27–35]. Chen at al. in 2020 proposed an algorithm to detect person falling based on OpenPose [36]. Considering that OpenPose cannot always output accurate coordinate positions of body parts, relative position and orientation between the camera and the person are diverse, OpenPose takes about 100 ms time delay to produce output, hence the algorithm calculates the ratios of several keypoints coordinate positions and increase discrimination conditions and process video frames with a time interval of 0.25s. Specifically the discrimination conditions consist of the speed of the descent at the center of the hip joint, the angle between the centerline of the body and the ground, the width to height ratio of the human body external rectangular. The algorithm achieves 98.3% sensitivity, 95% specificity, 97% accuracy. However, the algorithm mainly takes advantage of spatial coordinate information but nearly none of temporal information to improve false alarm, and the paper discusses little about response time which is critical for this application scenario.
More Insights
Through above literature analysis, difference insights can be extracted and applied to future research. The key learned lessons are summarized as below:
We can design a NN architecture with different functionality modules. In this way, on one side, we can enhance interpretability of DNN and make users and make readers capture the purposes of functionality NN modules, on other side, parallel branches with similar structure but different functions can enhance the whole DNN learning capability.
We can design a NN architecture by utilizing sequential stages. Through longer sequential stages and training the whole model by optimization stage after stage, we will get the model with higher accuracy while avoid gradients vanishing or explosion during training. It is also convenient to prune some stages when the model deployed on the resource limited devices.
Domain knowledge can be introduced to improve the model accuracy and reduce the model complexity. In OpenPose, the PAFs generation is based on width and length of limbs, and the bipartite matching algorithm is proposed after analyzing standard human skeleton, so that it is not necessary to solve NP hard K-partite graph problems.
To improve the performance of the current model, we can utilize the architecture redesign (cascade and parallel, sequential order), operation kernel redesign, functionality enhancement (bring in more relevant information), transfer learning.
DNN heavily relies on the training data. Appropriate annotated dataset can produce well performed models, otherwise resulting in garbage in and garbage out. Moreover, imbalanced dataset has negative impact on the models, which means the models are sensitive to the parts where abundant data are located.
In order to deploy the model on edge devices, we need to cut down the model complexity by several techniques including: using lightweight networks, merging similar functions or similar structure computation modules, compromising accuracy to reduce computation intensive modules, and others (quantization, pruning, knowledge distillation, etc.)
2D detection is limited by 2D images and unavailable temporal information. DNN cannot guarantee high performance when human visual inspection finds them difficult to process. If it is possible to introduce more information, such as 3D information from different angles cameras or temporal information by taking continuous video, the detection accuracy may increase.
Get Henry Heng LUO’s stories in your inbox
Join Medium for free to get updates from this writer.
For some certain application scenarios such as falling detection, rather than using 2D image processing output values, but using ratios by these values can be supplementary to 2D detection inefficacy. Therefore post model processing is quite vital.
References
1. Zheng, C.; Wu, W.; Yang, T.; Zhu, S.; Chen, C.; Liu, R.; Shen, J.; Kehtarnavaz, N.; Shah, M., Deep learning-based human pose estimation: A survey. arXiv preprint arXiv:2012.13392 2020.
2. Cao, Z.; Simon, T.; Wei, S.-E.; Sheikh, Y. In Realtime multi-person 2d pose estimation using part affinity fields, Proceedings of the IEEE conference on computer vision and pattern recognition, 2017; pp 7291–7299.
3. He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. In Mask r-cnn, Proceedings of the IEEE international conference on computer vision, 2017; pp 2961–2969.
4. Yan, S.; Xiong, Y.; Lin, D. In Spatial temporal graph convolutional networks for skeleton-based action recognition, Thirty-second AAAI conference on artificial intelligence, 2018.
5. Sun, Y.; Zheng, L.; Yang, Y.; Tian, Q.; Wang, S. In Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline), Proceedings of the European conference on computer vision (ECCV), 2018; pp 480–496.
6. Sun, C.; Shrivastava, A.; Singh, S.; Gupta, A. In Revisiting unreasonable effectiveness of data in deep learning era, Proceedings of the IEEE international conference on computer vision, 2017; pp 843–852.
7. Sun, K.; Xiao, B.; Liu, D.; Wang, J. In Deep high-resolution representation learning for human pose estimation, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019; pp 5693–5703.
8. Papandreou, G.; Zhu, T.; Kanazawa, N.; Toshev, A.; Tompson, J.; Bregler, C.; Murphy, K. In Towards accurate multi-person pose estimation in the wild, Proceedings of the IEEE conference on computer vision and pattern recognition, 2017; pp 4903–4911.
9. Zhao, Z.-Q.; Zheng, P.; Xu, S.-t.; Wu, X., Object detection with deep learning: A review. IEEE transactions on neural networks and learning systems 2019, 30 (11), 3212–3232.
10. Simon, T.; Joo, H.; Matthews, I.; Sheikh, Y. In Hand keypoint detection in single images using multiview bootstrapping, Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017; pp 1145–1153.
11. Fang, H.-S.; Xie, S.; Tai, Y.-W.; Lu, C. In Rmpe: Regional multi-person pose estimation, Proceedings of the IEEE international conference on computer vision, 2017; pp 2334–2343.
12. Xiao, B.; Wu, H.; Wei, Y. In Simple baselines for human pose estimation and tracking, Proceedings of the European conference on computer vision (ECCV), 2018; pp 466–481.
13. Mehta, D.; Sridhar, S.; Sotnychenko, O.; Rhodin, H.; Shafiei, M.; Seidel, H.-P.; Xu, W.; Casas, D.; Theobalt, C., Vnect: Real-time 3d human pose estimation with a single rgb camera. ACM Transactions on Graphics (TOG) 2017, 36 (4), 1–14.
14. Baltrusaitis, T.; Zadeh, A.; Lim, Y. C.; Morency, L.-P. In Openface 2.0: Facial behavior analysis toolkit, 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), IEEE: 2018; pp 59–66.
15. Wang, T.-C.; Liu, M.-Y.; Zhu, J.-Y.; Liu, G.; Tao, A.; Kautz, J.; Catanzaro, B., Video-to-video synthesis. arXiv preprint arXiv:1808.06601 2018.
16. Zhou, X.; Wang, D.; Krähenbühl, P., Objects as points. arXiv preprint arXiv:1904.07850 2019.
17. Xiang, Y.; Schmidt, T.; Narayanan, V.; Fox, D., Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint arXiv:1711.00199 2017.
18. Mahajan, D.; Girshick, R.; Ramanathan, V.; He, K.; Paluri, M.; Li, Y.; Bharambe, A.; Van Der Maaten, L. In Exploring the limits of weakly supervised pretraining, Proceedings of the European conference on computer vision (ECCV), 2018; pp 181–196.
19. Chen, Y.; Wang, Z.; Peng, Y.; Zhang, Z.; Yu, G.; Sun, J. In Cascaded pyramid network for multi-person pose estimation, Proceedings of the IEEE conference on computer vision and pattern recognition, 2018; pp 7103–7112.
20. Tulyakov, S.; Liu, M.-Y.; Yang, X.; Kautz, J. In Mocogan: Decomposing motion and content for video generation, Proceedings of the IEEE conference on computer vision and pattern recognition, 2018; pp 1526–1535.
21. Newell, A.; Huang, Z.; Deng, J., Associative embedding: End-to-end learning for joint detection and grouping. arXiv preprint arXiv:1611.05424 2016.
22. Cao, Z.; Hidalgo, G.; Simon, T.; Wei, S.-E.; Sheikh, Y., OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. IEEE transactions on pattern analysis and machine intelligence 2019, 43 (1), 172–186.
23. Osokin, D., Real-time 2d multi-person pose estimation on CPU: Lightweight OpenPose. arXiv preprint arXiv:1811.12004 2018.
24. D’Antonio, E.; Taborri, J.; Palermo, E.; Rossi, S.; Patanè, F. In A markerless system for gait analysis based on OpenPose library, 2020 IEEE International Instrumentation and Measurement Technology Conference (I2MTC), IEEE: 2020; pp 1–6.
25. Nakano, N.; Sakura, T.; Ueda, K.; Omura, L.; Kimura, A.; Iino, Y.; Fukashiro, S.; Yoshioka, S., Evaluation of 3D markerless motion capture accuracy using OpenPose with multiple video cameras. Frontiers in sports and active living 2020, 2, 50.
26. Slembrouck, M.; Luong, H.; Gerlo, J.; Schütte, K.; Van Cauwelaert, D.; De Clercq, D.; Vanwanseele, B.; Veelaert, P.; Philips, W. In Multiview 3d markerless human pose estimation from openpose skeletons, International Conference on Advanced Concepts for Intelligent Vision Systems, Springer: 2020; pp 166–178.
27. Ren, L.; Peng, Y., Research of fall detection and fall prevention technologies: A systematic review. IEEE Access 2019, 7, 77702–77722.
28. Mubashir, M.; Shao, L.; Seed, L., A survey on fall detection: Principles and approaches. Neurocomputing2013, 100, 144–152.
29. Koshmak, G.; Loutfi, A.; Linden, M., Challenges and issues in multisensor fusion approach for fall detection. Journal of Sensors 2016, 2016.
30. Delahoz, Y. S.; Labrador, M. A., Survey on fall detection and fall prevention using wearable and external sensors. Sensors 2014, 14 (10), 19806–19842.
31. Han, G.; Jiang, J.; Shu, L.; Niu, J.; Chao, H.-C., Management and applications of trust in Wireless Sensor Networks: A survey. Journal of Computer and System Sciences 2014, 80 (3), 602–617.
32. Zhang, J.; Li, W.; Ogunbona, P. O.; Wang, P.; Tang, C., RGB-D-based action recognition datasets: A survey. Pattern Recognition 2016, 60, 86–105.
33. Tomii, S.; Ohtsuki, T. In Falling detection using multiple Doppler sensors, 2012 IEEE 14th International Conference on e-Health Networking, Applications and Services (Healthcom), IEEE: 2012; pp 196–201.
34. Ke, S.-R.; Thuc, H. L. U.; Lee, Y.-J.; Hwang, J.-N.; Yoo, J.-H.; Choi, K.-H., A review on video-based human activity recognition. Computers 2013, 2 (2), 88–131.
35. Niveditha, P.; Manasa, S.; Nikhitha, C.; Gowda, H. R.; Natesh, M. In A Survey on Different Methodologies Involved in Falling, Fire and Smoke Detection Using Image Processing, International conference on Computer Networks, Big data and IoT, Springer: 2018; pp 378–384.
36. Chen, W.; Jiang, Z.; Guo, H.; Ni, X., Fall detection based on key points of human-skeleton using openpose. Symmetry 2020, 12 (5), 744.
