A Robust and Accurate Deep Learning based Pattern Recognition Framework for Upper Limb Prosthesis using sEMG

Surface Electromyography or sEMG is a significant physiological parameter that is used in the characterization of neuromuscular pathologies, muscle computer interfaces (MCI) such as bio robotics and human motor control based applications [1, 2]. In general, sEMG signals are used in EMG-PR based systems for pattern recognition and subsequently supervised learning is utilized for pattern classification.

It is considered that employing an instantaneous EMG signal to a pattern recognition model is not effective for pattern classification. Classical EMG motion recognition technique requires processing steps such as: (i) Data acquisition and pre-processing, (ii) feature extraction, (iii) dimensionality reduction, and (iv) motion identification or classification[5], [6], [7]. Due to non-stationary and stochastic property of the EMG, the instantaneous measures of the signal are inappropriate for the training of the traditional machine learning algorithm [20]. Therefore, in the feature extraction, a short time window (typically between 100-250ms) is applied to produce additional stability and enhance information concentration [9]. This step is required before a classification algorithm to compute the output on raw EMG signal. Several time, frequency, and time-frequency domains have been investigated for feature extraction in recent years. One of the renowned feature extraction techniques consists a set of MAV (mean absolute value), SSC (number of times slope sign change), ZC (zero crossings), WL (waveform length) named as Hudgins time-domain feature set [10], [11]. In recent years, short-time Fourier Transform (STFT) and continuous Wavelet transform (CWT) have been reported for pattern detection of sEMG data. [12]. Time derivative moment and fused wavelet packet transform-based features have been also proposed and observed a significant enhancement in pattern recognition performance [13], [14]. However, handcrafted features have played a significant role in EMG based pattern classification so far, deep learning has recently begun to showcase greater achievements than hand-engineered features [15], [16]. Furthermore, the existing system for grasping motions only satisfy to a certain extent human-centric activities, These include the necessity for a large number of functions, faster reaction/execution times, and control system intuitiveness. Fig. 1 depicts the functional diagram of the deep learning-based motion classification technique.

Deep learning algorithms have grown more prevalent in recent years because they are unrivalled in their capacity to automatically learn discriminate features from large volumes of data. The basic goal of deep learning is to shift the emphasis from human feature extraction to automated feature learning [17]. However, neural network-based architecture has been widely used for EMG-based motion recognition. Deep learning algorithms have had a significant influence on pattern recognition employing high-end computer workstations in the Big Data era. Convolutional neural networks (CNNs) are a type of deep neural network that has garnered a lot of traction in the field of pattern recognition. End-to-end learning models can be solved using CNN-based architecture. The spectral representation of EMG signals, such as Short-time Fourier transform (STFT) and Wavelet transform-based images, have been investigated to be fed to these neural networks. [18, 19]. Atzori et al. [20] has explored CNN with different classification techniques on NinaPro Data sets. The average classification accuracies of 66.6% and 60.30% have been achieved on NinaPro sub-database 1 and 2. In the research [21], the classification accuracy of 68.98% has been achieved for the NinaPro DB5 dataset (18 hands/wrist gestures), and this experiment was conducted over 10 subjects on a single Myo Armband. The main factor for accurate recognition when dealing with deep learning algorithms is the amount of training data available. The deep learning-based methods require the signal to image conversion [5], the large amount of data, and a long time for the training[22]. The translation deep learning technique to deploy for prosthesis application is still challenging. This requires large amount of data and training time. To address these problems, the goal of this work is to shorten training time and improve the performance of the CNN-based architecture for EMG-PR applications. The following are the primary contributions of this work: I a novel Deep Learning-based Pattern Recognition (DLPR) framework that uses a CNN to improve the performance of sEMG-based gesture recognition, (ii) data classification was done on five open source challenging and heterogeneous data sets that included both intact and amputated participants. (iii) analysis of classification accuracy in relation to DASH (disability of the arm, shoulder, and hand), (iv) training time of the suggested mode for all data-sets, and (v) comparison of the proposed work with state-of-the-art methodologies.

This paper is organized as follows. The information about experimental data has been discussed in Section II. Section III presents the proposed DLPR architecture for EMG-PR based bio-robotics and prosthesis. The result is emphasized in section IV. Section V covers the discussion about the result. The conclusion of this work is covered in Section VI.

The performance of the proposed DLPR architecture has been validated on five publicly accessible, scientific, and benchmark datasets NinaPro dB1, NinaPro dB2, and NinaPro dB3[1] (http://ninapro.hevs.ch/), upper limb motions with force variations[23] (DB4), and variant limb positions[24] (DB5) (https://www.rami-khushaba.com/electromyogram-emg-repository.html). The demographic information of data-set-1 and data-set-2 can be found in Table I. In the NinaPro DB1, an EMG signal of a total of 27 subjects has been recorded with otto Bock 13E200 electrodes and a filter of 90 to 450 Hz has been applied. This data-set contains 52 upper-limb motions and these motions have been divided into 3 categories: (i) 12 Basic movements of fingers, (ii) 8 Isometric, isotonic hand configuration with 9 basic wrist movements and (iii) 23 Grasping and functional movements.

EMG data of 40 subjects have been acquired in the NinaPro DB2 with Delsys Trigno wireless system and a band-pass filter of 20 to 450 Hz has been utilized. This data-set was acquired for 17 fingers and wrist movement, 23 grasping and functional motions, and 9 motions for different force pattern applied on fingers. NinaPro DB3 has also listed into 3 categories same as NinaPro DB1 for transradial amputees. In this work, 10 subjects have been taken into consideration from NinaPro DB1 and NinaPro DB2. Subsequently, 5 transradial amputees have been selected from the NinaPro DB3. The details of the selected subjects have been presented in Table 2. In the DB4, a total of 9 subjects comprises two congenital and 7 traumatic have participated. The movements are: (i) Thumb flexion, (ii) Index flexion, (iii) Fine pinch, (iv) Tripod grip, (v) Hook grip (hook or snap), (vi) Spherical grip power. Each subject is requested to use his intact-hand to imagine the required movement with the selected force level. Moreover, they utilized Visual Feedback (VF) from the system screen to see the EMG signals of all channels to make it helpful for them to produce the desired force. For each of the 6 grip patterns, the amputees produced 3 force levels: low, medium and high. In every phase of different force levels, 5 to 8 times were acquired for each amputee where each activity had a holding phase of 8-12 seconds. Further, in DB5, the same movements were performed at different limb positions for different EMG patterns. Total 11 subjects including 9 males and 2 females, of 20 to 37 years of age were selected to carry out 8 movements. A set of 8 hand motions, including  (i) wrist flexions, (ii) wrist extensions, (iii) pronation, (iv) supination, (v) power grip, (vi) pinch grip, (vii) open hand and (viii), rest, were performed at various limb locations by each subject.

Raw myoelectric signal for EMG-PR application is ineffective because the raw EMG signal is non-linear, non-stationary, and stochastic in nature [25]. This property of EMG signal is because of continuous variation in motor unit recruitment and arbitrary manner in which these motor unit action potentials are superimposed as shown in Fig. 2.

The time differential property of the Fourier Transform suggests that the $n^{th}$ derivative of a signal can be determined by multiplying the frequency spectrum of a signal by $k$ raised to the power $n$. It is specified in the equation (1).

Where $\sigma(j)$ signal in time domain, $\triangle^{n}$ represents the $n^{th}$ derivative of signal, and $\chi\left[k\right]$ is the frequency transform of the signal.

The power spectral moments are used to pre-process the signal before feeding input to the neural network. The moment $\mu$ of $n^{th}$ order for window length $T$ is defined as in equation (2).

The IOS (integral square) of the segmented signal can be defined as its power, as the parseval ($\eta$) theorem suggests as presented in the equation (3).

As per the equations (1), (2), and (3) the $0^{th}$, $2^{nd}$, and $4^{th}$ order moments are defined as follows:

In the same way as described previously, the fourth order moment is depicted in equation (6).

The NPs (number of peaks) and ZCs (zero crossings) of a stochastic process can be calculated using their spectral moments [26], [14]. This is defined in equations (7) and (8).

The square version of NPs and ZCs has been used to simplify equations (7) and (8), as illustrated in equations below.

The sEMG signal is passed via two functions in the recommended architecture: I MPP ($psi$), which is the multiplication of peaks and power, and (ii) MZP ($phi$), which is the multiplication of zero crossings and power. This phase stabilises the EMG signal and reduces the size of the training data set while maintaining neuronal information. The peaks of these parameters include frequency information, whereas the zero order moment includes time information without any time to frequency transformation.

As shown in Fig.3 the suggested CNN structure is made up of six blocks: three convolution layers, two fully connected layers, and one softmax layer for classification. The initial convolution layers, which comprise 128 filters with a kernel size of 7, are used to feed pre-processed data. The output of the first convolution layer is 18 x 128. A 14x128 frame is created by the second convolution layer. Following that, the design has been updated to accommodate a size 2 max pool, resulting in a 7x128 output. The third convolution layer, which comprised of 64 filters with a kernel size of 3, has also been employed. The output of the third convolution layer is 5x64. After that, global average pooling is employed between the third and fourth fully connected (FC) layers of size 512. The softmax layer is then added, followed by an FC of 128. The Adam optimization strategy has been used to apply batch normalisation and a ReLU non-linearity after each block. The mathematical formulation for layer-1 can be written as follows in equation (13).

where $\alpha_{m}^{l}(j)$ belongs to neuron $j$ of map m in layer $l$. It also shows the multiplication between the weighted values and input neurons.The filter used in the neural network is $w_{k}\left(l,m,i\right)$. layer, map, channel/kernel and the position are defined by $l$, $m$, $i$ and $k$ in the kernel respectively. The bias in the layer-1 for the neuron j is indicated by $b^{l}(j)$. For layer-2 and layer-3 the mathematical expression is defined as equation (14).

The number of kernels are denoted by $N_{kernel}^{l}$ in layer $l$. $M_{i,j}$ belongs to the components i, j (neuron j in-kernel i) of the feature maps produced following convolution from the preceding layers. Finally, 2 fully connected layers (FCs) are used and the softmax layer of size 23 is employed.The classification accuracy metric has been considered for the performance evaluation as denoted in equation 15 and each data-set has been divided into two parts: (i) training data (60% of the total data), (ii) testing data (40% of the total data).

The performance of the proposed DLPR based framework using DNN is validated using five different data-sets. The window size of 100 samples with a shifting of 10 samples has been selected when the proposed framework is used for NinaPro DB1 and a window size of 300 samples with shifting 50 samples has been considered for other remaining data-sets. The first three data-sets (NinaPro DB1, NinaPro DB2, NinaPro DB3) have been tested using three different batch sizes (50, 100, and 150). The training and testing classification accuracies have been shown in Fig. 4 (a), (d), and (g) for 50 epochs. The classification accuracies for the NinaPro DB1 of 91.11%, 90.95%, and 88.45% have been observed for the batch size of 50, 100, and 150 respectively, as shown in Fig. 4 (a), (b), and (c) . For the NinaPro DB2, the classification accuracy of 88.43%, 89.45%, and 87.49% have been seen for the three batch sizes, presented in Fig. 4 (d), (e), and (f). The classification accuracy of 79.81%, 81.67&, and 80.44% have been exhibited when NinaPro DB3 has been taken into consideration in Fig.4 (g), (h), and (i) for the same batch sizes. Pattern classification of data-set 4 shows 95.67% whereas pattern recognition rate for data-set 5 increased to 99.12% after 20 epochs as denoted in Fig. 4 (a) and (b) respectively.

The proposed DLPR method exhibits significant improvement in classification performance with respect to classical machine learning techniques and with spectrogram images based on CNN architecture. In this paper, NinaPro DB3 has been considered for evaluating pattern recognition performance on amputated subjects as shown in Fig. 5. All the amputated subjects have different physical properties in terms of DASH score, remaining forearm, and phantom limb sensation. Subject 11, who had lost his limb due to cancer and having DASH score 12.5 achieved a classification accuracy of 92.26%. Subject 2 (DASH=15.18) and subject 8 (DASH=33.33) have achieved the lowest classification accuracy of 80.34% and 80.92% respectively. Subject 9 has a DASH score of 3.3 and has achieved 86.45% classification accuracy. This shows that pattern recognition performance has been boosted remarkably when NinaPro DB3 has been taken into consideration. Furthermore, this work is compared with literature and is also summarized in Table IV. In the research [20], three data-sets were considered and classification accuracies of 75.32% (RF), 75.27% (RF), and 46.27% (SVM) were achieved for NinaPro DB1, NinaPro DB2, and NinaPro DB3 respectively. Paper [27] reported 78.71% classification accuracy for the NinaPro DB2 (healthy subjects and 49 classes) using CNN. Total 2 data-sets Hand/writ and NinaPro DB5 (18 grasping activities) have been considered in the research [21]. The classification accuracy of 98.31% and 68.98% have been achieved for these 2 data-sets respectively. Total 10 upper limb activities (Dynamic and Air) were classified using LSTM (low short term memory) network and classification accuracy of 89.28% was obtained. Data-sets (NinaPro DB1 and NinaPro DB2) have been used and a multi-view deep learning framework was employed. In this research, the pattern recognition rate 88.20% and 83.70% were achieved for NinaPro DB1 and NinaPro DB3 respectively. However, data with force variation and position variation have not been considered. Proposed DLPR architecture has been examined for 5 different types of scientific and benchmark data-sets and the pattern classification rate is significantly boosted for all healthy and amputated subjects. At the initial stage, the raw EMG signal has been passed through two parameters (MZP and MPP) which reduces data size for the training with minimal loss of neural information. These parameters contain information regarding time as well as frequency present and this helps to train a highly accurate classification model. The proposed model shows robust performance by showing significant enhancement in classification accuracy in all kinds of data-sets.

Furthermore, training time for different data-set have been evaluated as represented in Table V. This demonstrate that this model only takes few minutes time to get the high classification accuracy.

The proposed DLPR-based system is a robust, fast, and precise deep learning-based framework for classifying diverse gripping and functional activities for upper limb prosthesis applications. The investigation is carried out on five distinct types of standard and benchmark data-sets, comprising healthy and amputee people. This proposed approach is used as a pre-processing step in the primary phase to make input information-rich and stationary. With current equivalents, considerable functional development has been noticed in terms of classification accuracy of up to 25% and time complexity is decreased significantly. Classification performance of amputees with a high DASH score is also improved. In future work, this techniques will be translated into DSP processor for real-time EMG pattern classification.

The authors would like to acknowledge the support of the Ministry of Human Resource and Development, India, Bionics and Intelligence Lab. Jaipur, India, and Dr. Rami N Khushaba and Dr. Ali H. Al-Timemy for data-sets 4 and 5.