Grasp-Oriented Myoelectric Interfaces for Robotic Hands: a Minimal-Training Synergy-Based Framework for Intent Detection, Control and Perception

In this paper, a general framework for myoelectric humanrobot interfaces for grasp-oriented tasks is presented. In particular, the aspects of intent detection, control and perception are all three integrated in the framework. Particular attention is devoted to provide a minimal-training procedure for the user, and to convey sensory feedback information from contact force sensors in a compact manner, by means of a single vibration motor to foster practical real-world usage of the interface. Specifically, in the proposed framework, the user’s motor intention is estimated from electromyographic measurements exploiting the concept of muscular synergy, together with an unsupervised learning approach that enables a short and simple algorithm calibration. Then, the postural synergy concept is exploited in an encoding-decoding fashion, in order to communicate the user’s hand closure level to the robotic hand, i.e. exploiting kinematic dimensionality reduction. Moreover, a rough estimation of the internal grasping force applied by the robotic hand is exploited for the generation of a sensory substitution feedback, which is applied to the user by means of vibrotactile stimulation. After describing the general structure of the proposed framework, we show a practical design for a real interface implementation. Finally, experimental tests with 4 näıve subjects are reported demonstrating the actual effectiveness of the approach – in terms of mean and standard deviation of grasp strength reference trackings – in fine object grasping tasks.


Introduction and Research Context
Robotics is currently experiencing a growing demand for a closer cooperation, interaction and integration with humans. Driven by this need, in the recent years a new generation of robots characterized by natural and intuitive human-machine interfaces was developed, in which the design steered towards human-inspired solutions, as a road to replicate the human ability and flexibility in performing motor tasks. A key element in the general framework of Human-Robot Interfaces (HRIs) is the so-called human-in-the-loop integration, where the human-machine interaction is a fundamental and constitutive element of the system design.
To enable a natural interaction between humans and robots, a major challenge is to leverage on signals acquired from the human body to communicate with the machines. Surface skin ElectroMyoGraphic (sEMG) signals are widely popular in several scenarios, ranging from rehabilitation to robot control, and in particular they are often exploited as an interface between human subjects and robotic hands. With this respect, it is worth noticing that the human hand shows remarkable capabilities in object manipulation, whereas, in contrast, robotic devices are chracterized by poor performance in fine physical interaction with objects and their control is still a big challenge in the scientific community [1]. However, the configuration space dimension of the human hand can be reduced -considering a set of common grasping tasks -thanks to the concept of postural synergies [2], according to which the grasp synthesis for a large set of daily-life objects is achievable by a linear combination of a limited set of dominant hand postures, commonly referred as postural synergies or eigenpostures.
Aiming to reproduce the human's intention on a robotic device, machine learning techniques are widely used to improve the correlation between muscle activity measured through sEMG on a human subject and the corresponding motion commanded to a robotic hand. Several authors explore different classifiers, such as neural networks [3], Linear Discriminant Analysis (LDA) [4] or Support Vector Machines (SVMs) [5]. However, classifier-based approaches present several drawbacks, such as long training periods, large demand of human time for data labeling and limited set of implementable task or action classes, resulting in limited affordability and flexibility of the human-machine interface.
To overcome these limitations, several supervised machine-learning-based regression algorithms have been implemented by the research community. The advantage of this method lies in the possibility of continuously controlling the DoF, so that ambiguities in input signals do not lead to totally unwanted results as in case of classifiers. However, despite several positive outcomes, regressionbased ME HRIs presents several reliability issues. This is likely due to two main assumptions on which this method relies: invariance of the sEMG signals with respect to subjects and time [6] and accuracy/reliability of the data labeling for the algorithm training. These conditions are hardly satisfied in practice -both for intact users and rehabilitation applications -resulting in degraded performance and instability in the robotic hand interface. Additionally, the knowledge gained in the past decades about the neurophysiological processes underlying natural movements, such as muscular synergies, is not exploited in the supervised machine learning framework. For these reasons, in the present work we propose an ME interface for robotic hands based on unsupervised machine learning technique in which data regression is implemented by means of a matrix factorization approach, thanks to the modelling of the human's neuro-motor system combined with muscular and postural synergy concepts. In this way, the training proce- dure and the data labeling is not necessary, and only an extremely simple and fast calibration data acquisition is required.
Another important aspect related to the use of myocontrolled robotic hands for the execution of grasping tasks is the availability of an artificial sensory feedback related to the grasp strength. The most common methods are patterned electrotactile, mechanotactile, or vibrotactile stimulation of the skin, in accordance with the general approach known as sensory substitution [7]. In the present work, we focus on the vibrotactile feedback, i.e. the use of vibration motors to deliver skin mechanical stimulations. Note that vibrotactile feedback is not new at all, since it has been already studied in several works [8][9][10]. However, despite the fact that many works answered the question if vibrotactile force feedback can help the user in executing grasping operations (and the problem is still open [11]), in this work -since a grasp-oriented interface for practical applications is concerned -we will focus on how to enable the user to precisely reproduce grasp strength reference values exploiting vibrotactile feedback with a myocontrolled robotic hand. For this reason, the main contribution of this work consists in the presentation of a complete myoelectric human-robot interface framework for robotic hands control, in which the user provides commands to the robotic device, equipped with contact force sensors mounted on the fingertips, and receives an information that synthesizes the interaction force during grasping via a simple vibrotactile device. In this framework, the detection, the control and the perception aspects are all considered together with the design and implementation issues in a synergy-based approach. Other important features we embedded in the proposed framework are the minimal training effort, the independent force regulation of the fingers to adaptation the grasp to objects, a sensory substitution approach based on contact force sensors measurements for the use with EMG signals.
In the following, the proposed framework is presented together with its practical implementation. Moreover, the experimental evaluation of the framework is carried out and the results are reported and analyzed.
2 Synergy-based framework for myoelectric HRI

General Structure Overview
The framework presented in this work for grasp-oriented myolectric interfaces is depicted, in its structure, in Fig. 1. In the figure, the different blocks composing the bidirectional chain of the human-robot's interaction are represented, subdivided in the areas of (i ) Human, (ii ) Myocontrol, (iii ) Robotic Hand Controller, (iv ) Robot and (v ) Sensory Substitution Feedback.
The Human area is the interface zone in which the neuro-motor information is gathered from the user by means of measurements related to some physiological activity -in particular, in this framework we specifically consider the sEMG signal. This general concept is discussed from a design point of view and implementation case in Subsec. 2.2, in terms of electrodes, data acquisition and signal processing. At the same time, also the interfacing to the sensory feedback stimulation is here located, as reported for the real case scenario in Subsec. 2.4. In the Myocontrol area, the human's intent interpretation takes place: in the proposed framework this is based on a modelling of the sEMG-wise intent generation, lying on the exploitation of the muscular synergy and neural drive concepts, with the specific aim of applying unsupervised learning techniques in order to have minimal training sessions of the myocontrol algorithm by the user of the interface. Here, also the concept of postural synergy is exploited, in order to encode the human kinematic information in a neuro-physiologically compact form (i.e.: dimensionality reduction). Accordingly, the Robotic Hand Controller area is composed by a synergy-based decoder of the motion, that in this way can be provided as input for the lower-level servomotor controller of the anthropomorphic grasping device, providing appropriate torques according to the requested behaviour -details of this concepts in a practical sense are illustrated in Subsec. 2.3. The Robot area includes the robotic hand, which is equipped with contact force sensors both for sensory feedback generation and for servomotor controller in order to adapt to different object shapes (hardware and design aspects are reported in Subsec. 2.3). Then, the interaction forces exchanged between robot and external world are interpreted and modulated within the Sensory Feedback Substitution area, in which the signals detected by the contact sensors are translated into a meaningful stimulation for the human, which is provided by a vibrotactile actuator for the feedback delivery (the practical implementation is reported in Subsec. 2.4).

Factorization-based Myocontrol
sEMG Acquisition Setup and Pre-Processing The sEMG signal acquisition hardware is based on the Cerebro board [12]. The data are acquired at 1 kHz and streamed to a nearby computer using the onboard Bluetooth interface. The placement of the electrodes is conceived to cover the portion around the forearm in proximity of the Flexor Digitorum Superficialis and the Extensor Digitorum Communis muscle bellies. As can be seen in Fig. 2, 16 disposable surface skin electrodes are arranged to form an armband composed by 8 pairs of differential electrodes uniformly distributed, obtaining in this way 8 sEMG signals. Then, a standard online filtering procedure is implemented, consisting of: (i) a 50 Hz notch filter; (ii) a 20 Hz high-pass filter; and finally (iii) the root mean square (RMS) value of the signal is calculated over a 200 ms window.

Synergy-Based Model of Human's Hand Closure Motion Control
Let us consider the RMS value of the online 8-channel sEMG acquisition (see Sec. 2.2) E on (t) ∈ R 8×1 . We suppose that at each time instant this multidimensional signal can be expressed as the product of the muscular synergy matrix S M ∈ R 8×2d and the neural drives U on (t) ∈ R 2d×1 [13], where d is the number of DoF activated by the muscles that generate the muscle activity E on (t): (1) Note that, while E on (t) is known, in eq. (1) S M and U on (t) are the unknowns. The concept of postural synergies is exploited in order to consider the overall opening and closing of the human hand as a unique DoF, in particular the only one activated by the muscles generating the muscle activity E on (t). Moreover, such movement can be considered as a single synergistic motion, given by one specific postural synergy. Therefore, according to this consideration, we assume E on (t) expresses the muscle activity during the regulation of such single synergistic DoF, which means that d = 1. With these assumptions in mind, it is possible to consider two antagonistic actions, namely "generalized" flexion and extension, for the actuation of that synergistic DoF, i.e. the hand closure. Similarly, we can also consider two neural drives that activate such antagonistic actions. In this way, exploting the antagonistic actuation concept [14], it is possible to write the matrices in eq. (1) as where s e , s f ∈ R 8×1 are the extension and flexion components of the muscular synergy matrix and u e (t), u f (t) ∈ R are the extension and flexion components of the neural drives. Equations (2) and (1) (2) and (1), E off can be expressed as with where U off ∈ R 2×n is the matrix of the offline neural drives composed of the extension and flexion components u e , u f ∈ R n×1 (note that they are not a function of time in this case). Taking into account (3), and on the light of the a priori knowledge of a unique synergistic DoF activated during the calibration process, it is possible to solve (4) with respect to S M and U off . Consequently, a factorization algorithm can be applied to compute the S M . In this way, once the muscular synergy matrix is estimated, it is possible to use its pseudo-inverse S + M to online compute the neural drives of the human hand opening/closing motion as U on (t) = S + M E on (t). In order to obtain a proportional control signals for the grasp control, the concept of the antagonistic actuation model [14] is exploited. Two antagonistic actions are considered, representing two groups of flexor and extensor hand muscles located in the forearm, that control the synergistic DoF and the stiffness level of the hand. According to this concept, the online estimated neural drives are linearly combined to obtain the closure hand reference ρ ref (t) control signals as

Human-like Robotic Hand Controller
The UB Hand: Structure and Sensory equipment The University of Bologna Hand (UB Hand) is a robotic hand with an anthropomorphic structure [15,16], see Fig. 3(a). Each one of the 5 fingers, characterized by 3 DoF, is fully actuated via 5 servomotors (Dynamixel RX-24F) located in the forearm whose motion is transmitted by inelastic tendons, which are properly coupled and routed in the finger structure. Furthermore, the UB Hand is equipped with three semi-spherical contact force sensors (OMD-20-SE-40N Optoforce [17]), able  to detect the magnitude of forces acting along the x, y and z directions as shown in Fig. 3(b), namely f x , f y , f z . The three sensors are mounted on the fingertip of the thumb, index and middle fingers by means of a 3D-printed support (see Fig. 3.) In this work, the norm of the force vectors measured the sensors is used.
They are denoted as Synergy-based Hierachical Controller In order to control the overall robotic hand, a proper mapping between the n m tendons actuating the joints, each one driven by a servomotor, and the n J joints of all the hand fingers exists. This control architecture is characterized by a three-level hierarchical structure, with three different levels: motor, joint and synergy. At the lowest level, a stiffness control has been implemented [18]. Therefore, the Dynamixel servomotors are actuated according to where τ m ∈ R nm is the vector of motor torques, K m ∈ R nm×nm is the diagonal matrix defining the motor stiffness, and θ ref m , θ m ∈ R nm are the motors reference and the actual position, respectively. At the joint level, the mechanical configuration of the tendon-based transmission system defines a linear mapping H ∈ R nm×n J between joint and motor configuration variables (for more details refer to [15,16]). This mapping can be used to deduce the reference angle of the motors corresponding to a given joint configuration θ ref By combining (6) and (7) it is possible to deduce the stiffness relation induced at the joint level by the motor control: where K J = H T K m H ∈ R n J ×n J defines the joint stiffness matrix. Finally, at the the higher synergy level, the joint reference configuration θ ref J is computed as reported in the following. Letθ The input variable σ(t), that appears in (9), is the myocontrol-driven signal obtained by applying a dead-zone to ρ ref (t) defined in (5), that is in which |f i | ∈ R + is the norm of the i-th finger force sensor,f is a constant threshold defining the minimum force value detected when the fingertip contacts an object,K is the control gain during free-space motions and γ is a scale factor determining the minimum value of K i that may occur during the grasping. The variation of the gain K i , that plays a central role in this control architecture,   is motivated by a twofold reason: (i) for a given level of the myocontrol signal the motion of the finger that came into contact with the object is drastically slowed down, allowing the hand to adapt to the shape of the grasped object, and (ii) the capability of the user in finely modulating the grasp strength is highly increased, thanks to the possibility of providing small adjustment to the robotic hand configuration, see eq. (9). Finally, note that, for i = {R, P }, K i has been set to zero because in this study only thumb, index and middle fingers were equipped with tactile sensors and consequently only tripodal grasps have been considered. In Fig. 4(a) the value of K i (for i = {T, I, M }) as a function of |f i | are shown.

Sensory Substitution for contact detection
Vibrotactile Feedback In order to implement the robotic hand myocontrol, the user is provided with a vibrotactile stimulation produced by means of a vibration motor embedded on the wrist armband of the Noitom Hi5 glove [20] (see Fig. 3(c)), a wearable device for hand's finger motion tracking and feedback stimulation. Note that the motion tracking functionality of the glove is not used in this work, and only the vibration motor was applied to the wrist of the subjects during the experimental tests, as illustrated in Fig. 3. This vibration motor can be controlled via a Bluetooth interface at the maximum rate of 1 Hz by commanding the vibration duty cycle. Accordingly, the vibrotactile stimulus was obtained according to a pulse-width modulation strategy, in which the input signal ν ∈ [0, 1], hereafter called normalized feedback signal, determines the dutycycle of the vibration motor activation signal, as illustrated in Fig. 4(b).

Force Sensing Interpretation
The vibrotactile stimulus is provided to the user on the basis of the normalized feedback signal ν, which depends on an estimation of internal forces generated by the grasp. Note that with internal force the forces exerted by the robotic hand fingers that produce zero resultant forces on the object is meant. Mathematically, these forces are defined as the force vector which spans the null space of the so-called grasp matrix, typically denoted with G [21]. In precision tripodal grasps, the thumb fingertip is located in an opposite configuration with respect to the index and middle fingertips, and its contact force is therefore counteracted by the other two fingers and viceversa. Consequently, the force exerted by the thumb at the contact location represents an good approximation of the applied internal force. For this reason, its measurement was used to approximate the internal force, and the normalized force signal was computed as where c norm is a normalization constant and β denotes an error band in which the tracking off T is considered achieved (in our experiment β = 0.03, see Sec. 3.2). The combination of (12) with the duty cycle modulation of the vibration motor ( Fig. 4(b)) produces the vibrotactile feedback. Note that, since the vibrotactile feedback depends on the absolute value of the error between estimated and target grasp strength, during the experiment reported in the following subsection the user was assisted by a simple visual cue (on a display) informing only about the sign of the error, see Fig. 5, while the extent of the error was available solely through the vibrotactile stimulation.

Experimental tests
In order to assess the effectiveness of the proposed control architecture, an experiment is conducted involving four naïve able-bodied subjects. The test is performed in accordance with the Declaration of Helsinki. Figure 5 shows the setup arrangement used in the experimental session. The subjects are four men between 25 and 35 years old (namely S1, S2, S3, S4), and none of them experienced myocontrol before. During the experiment, each subject is seated in front of a desk equipped with the robotic Hand, described in Subsec. 2.3. A couple of differential sEMG electrodes are placed in proximity of the FDS and EDC muscles' belly (see Susec. 2.2), and the vibration motor described in Subsec. 2.4 is fixed to the forearm in a location close to the wrist. Initially, during the myocontrol training session, each subject is asked to fully open and close his hand two times while the sEMG signals are recorded. Then, the system automatically computed the pseudo-inverse of the muscular synergy matrix S + M and the scaling constants as described in Sec. 2.2. After the myocontrol calibration, the grasping strength regulation task starts. When the myocontrol is connected to the UB Hand, the subject receive the vibrotactile feedback according to the strategy reported in Sec. 2. In each trial, the subject is initially requested to keep the robotic hand fully open, until the experimenter hands the first object out. At this point, the subject is asked to close the robotic hand in order to grasp the object with the three fingers, see Fig. 3. Once the object is grasped, the experimenter release the object and the subjects had to regulate -by exploiting the proposed myocontrol interface -the grasp strength, according to three given reference levels (f T,1 = 0.2 N,f T,2 = 0.4 N,f T,3 = 0.6 N). Each of the grasping strength level has to be reached and maintained for a duration of approximately 100 s. Then the target level is changed during the grasp, i.e. without the subject releases the object. At the end, the experimenter vocally instructs the subject to release the object (i.e. open the robotic hand) and take the object back. Three different objects are used during the experiments: a box, a cylinder and a sphere (supplied in this order), whose dimensions and images are reported in Tab. 1. Three different grasping and regulation tasks are carried out for the three different objects. The subject rests 10 minutes after each task.

Experimental Setup and Protocol
The results of the experiment are reported in the following paragraph, using as metrics the mean and standard deviation of the grasp strengh regulation profiles over the subjects, and the related settling times as an index of success in the task execution. Figure 6 shows the grasping strength signal obtained during the regulation task of one of the subjects (S4) with the Box object. The graph at the top shows that after a transient the prescribed grasp strength -well represented by the thumb force |f T | -was successfully applied by the user regardless of the specific level. Note the tracking of a reference level is considered successfully achieved when |f T | enters within an error band equal to ±5% of the higher strength reference level (0.6 N), i.e. in the range [f T,i − 0.03,f T,i + 0.03], i = 1, 2, 3, and remains in  this range until a new reference level is provided or until the experiment ends. The settling time is defined as the time required by the user to successfully reach the desired grasp strength level. By focusing on the graph showing the hand closure level σ input in Fig. 6, it is possible to observe how this signal was intuitively regulated in order to increase (time instants around 0 s, 100 s, 200 s) or decrease the force (time instants around 30 s, 290 s). This is more clearly shown in the graph plotting σ, the myocontroldriven signal given by eq. (10), where the values inside the deadzone are set to zero. Finally, the bottom plot of Fig. 6 illustrates how σ is modulated by K gain : when |f T | <f (orange dashed line in the top graph of Fig. 6), K T assumes the highest value, see the case#1 in eq. (11), while when f T ≥f , i.e. contact with the object is established, the gain K T quickly decreases to its minimum level equal toK/γ according to case#2 of eq. (11). Accordingly, the signal K T σ becomes smaller and smaller during the interaction with the object, and therefore the user capability of modulating the robotic hand joint velocity references becomes more and more accurate.

Grasp Strength Regulation Results
The experiments highlight that the behaviour shown in Fig. 6 is replicated by all the subjects, since all of them successfully regulate the grasping force for all the considered reference levels. These results are shown in Fig. 7 in terms of mean and standard deviation values of the regulated grasp strength, computed over the subjects. Note that, even if the performances of the users during the transient are quite different, producing a remarkable variability of the force strength during this phase, the desired final value is achieved by all the subjects -in terms of succesful reference tracking, as defined according to the settled time perviously mentioned in this paragraph -, as shown by the mean and standard deviation of the regulated grasp strenghs reported in Fig. 7.

Conclusions
In this paper, the implementation and the experimental evaluation of a human machine interface based on a synergy mapping of sEMG measurements and vibrotactile feedback for the regulation of the grasping strength on an anthropomorphic robotic hand is reported. The experiments demonstrate that the proposed framework allow a fast adaptation to different subjects thanks to a simple and fast online calibration procedure. Moreover, the synergy-based sEMG mapping simplified the detection of the human intention, enabling the implementation of a reliable interface between the human and the robotic hand, at least for a simple tripodal grasping task. The experimental results highlighted also that, thanks to the vibrotactile feedback, all the subjects are able to properly regulate different values of desired grasping strength.
Despite the simplicity of the adopted feedback strategy, the results reported in this paper are very promising and pave the way toward the extension of the proposed approach to a wider set of tasks and more advanced and flexible feedback techniques. Therefore, future work will be devoted to the experimental validation of the approach by extending the set of considered grasps and in the improvement and evaluation of the proposed vibrotactile feedback, as well as on the investigation of alternative feedback strategies. Finally, a larger set of subjects will be engaged for future experiments, in order to increase the experimental protocol's statistical power and provide an analysis of the statistically significance level of the results for a proper discussion on the generalization of the reported outcomes.