Human movement augmentation is a rising field of research. A promising control strategy for augmented effectors involves utilizing electroencephalography through motor imagery (MI) functions. However, performing MI of a supernumerary effector is challenging, to which MI training is one potential solution. In this study, we investigate the validity of a virtual reality (VR) environment as a medium for eliciting MI neural activations for a supernumerary thumb. Specifically, we assess whether it is possible to induce a distinct neural signature for MI of a supernumerary thumb in VR. Twenty participants underwent a two-fold experiment in which they observed movements of natural and supernumerary thumbs, then engaged in MI of the observed movements. Spectral power and event related desynchronization (ERD) analyses at the group level showed that the MI signature associated with the supernumerary thumb was indeed distinct, significantly different from both the baseline and the MI signature associated with the natural thumb, while single-trial classification showed that it is distinguishable with a 78% and 69% classification accuracy, respectively. Furthermore, spectral power and ERD analyses at the group level showed that the MI signatures associated with directional movement of the supernumerary thumb, flexion and extension, were also significantly different, and single-trial classification demonstrated that these movements could be distinguished with 60% accuracy. Fine-tuning the models further increased the respective classification accuracies, indicating the potential presence of personalized features across subjects.
A rising field of research in human movement augmentation is that of brain-computer interface (BCI) systems, which bypass the need for physical action to control the supernumerary effector and allow communication between the brain and device1. EEG, in particular, is a popular technique used for BCI control due to its non-invasive nature and high temporal resolution2. This method can capture time-frequency domain features such as ERD in the alpha band (9–13 Hz) at the motor cortex, phenomena linked to motor execution (ME) and MI3, of which the latter is often utilized in BCI applications. Despite its popularity, one major limitation of EEG is its low spatial resolution, which hinders the reliability of control, sometimes to the point where it can negatively impact both the controllability and concurrency of the effector. Given this, many implementations of EEG-based augmentation by extension often rely on triggering movements using MI signatures of pre-existing actions, such as walking in a specific direction or grasping an object4,5,6, improving reliability of control at the expense of concurrency. However, addressing this limitation directly and without compromise is a possibility that can be explored. The motor cortex has been found to possess the ability to represent additional natural effectors and their movements through brain plasticity without impeding motor functions7,8, showing the potential of a representation for supernumerary effector control. A recent field of research explores mapping the control of a supernumerary effector to a distinct MI signature. For example, Penaloza et al. proposed a BCI system that controlled an augmented third arm through MI associated with the supernumerary effector while the natural arms were simultaneously engaged with a ball-balancing task9,10, indicating a concurrency between supernumerary and natural effector control. In addition, a 4-week longitudinal study performed by Liu et al. provided further evidence towards the feasibility of creating a distinct MI function that could be mapped to the control of a supernumerary thumb, achieving a within-subject classification accuracy of 70% in distinguishing between MI and rest periods11. While both of these works involved MI of the supernumerary effector, it remains unclear whether the MI performed by individuals was strictly associated with the supernumerary effector itself. There is a lack of direct comparison between the neural signatures associated with the MI of movements of the supernumerary effector and those of the natural one, which is necessary to verify distinctiveness. Furthermore, these studies highlighted the need for more reliability in the induced neural activations, as MI performance and clarity varied across subjects such that classification could only be performed on an individual level (i.e., within-subject classification). One potential solution to these limitations in reliability is MI training, a concept relatively novel to the fields of BCI and augmentation, but well-researched in the fields of rehabilitation and sports12,13,14. MI training methods use individual or combinations of different motor stimuli in a repetitive manner to enhance MI performance, often associated with improved motor performance15. These stimuli include ME, physically performing the task to be imagined; motor observation (MO), observing the task being performed by someone else; or MI, mentally rehearsing the task without physically performing it12. However, we find that there are differences that should be considered in MI training of a supernumerary effector, as opposed to MI training for movements that involve natural effectors. For instance, literature on MI training suggests that ME be interleaved with MI trials for better performance12. However, it is difficult to ask an individual to perform the movement of an effector that is completely foreign, thus MO could be considered as a close substitute13. Additionally, given that first person MO and MI is reported to induce a greater sense of embodiment and may involve different parts of the brain16,17, mediums that allow a first person MO of an unfamiliar effector, such as virtual reality (VR) or extended reality (XR), could be considered as a training environment. Furthermore, though not as essential in rehabilitation and sports, ensuring that motor stimuli are performed sequentially rather than simultaneously is crucial from a BCI perspective. This is due to differences in the respective neural activations of MO and MI14,18, as well as the effects they are induced in close tandem15. Considering that the goal of the method is to control a supernumerary effector using motor imagery, performing MO and MI in parallel could create different neural signatures from isolated MI, hindering the quality of EEG data. Thus, in this study, our objective is to examine the validity of VR as an environment in eliciting distinct neural signature for a supernumerary effector, in this case, a virtual supernumerary thumb, through a cross-sectional study, by creating an MI paradigm involving two separate (i.e., consecutive but not simultaneous) first-person motor stimulations, MO and MI. We take a systematic approach by a) confirming the presence of distinguishable ERD during the MI of the supernumerary thumb; b) verifying that there is a distinction between the MI associated with the supernumerary thumb and the natural thumb; and c) identifying directional control within the MI of the supernumerary thumb. We employ trial-averaged statistical analysis and deep learning classification at each of these steps as a mode of validation of the results on a group-level and single-trial basis, respectively. We employ leave-one-subject-out cross-validation, examining the ability to extract group-level MI features while also examining the potential presence of personalized features by fine-tuning the model on test-subject data15,19,20. Through this study, we contribute to the field in several ways. We first validate the VR as a method to induce MI neural activation for supernumerary effectors, promoting it as a potential MI training environment. We also show the distinction of the neural signature associated with the MI od the supernumerary effector against that of a natural effector of similar nature, which has not yet been performed in previous literature to the best knowledge of the authors. We also investigate signatures associated with the MI of flexion and extension of the supernumerary effector, providing insight on directional control of an augmented, BCI-based effector. Furthermore, we perform these classifications on a group level, as opposed to only on an individual basis, exploring the possibility of generalized classification models that can be potentially personalized through fine-tuning.
Twenty subjects were recruited for this experiment, comprising 14 males and 6 females. Of these, 18 were undergraduate students aged between 18 and 25 years and 2 were individuals between 30 and 50 years. All participants were right-handed, performed tasks with their right hand, and had normal or corrected-to-normal vision. Exclusion criteria were being under 18, left-handed, or having a history of traumatic brain injury, neurological disorders, or muscle atrophy. Participants provided informed consent for their participation as per the Institutional Review Board (IRB) ethics, with the study being approved by the New York University Abu Dhabi’s IRB (HRPP-2023-60), in accordance with the Declaration of Helsinki. Participants received subsistence allowance for their participation. Four participants were excluded due to poor electrode connections or cap misalignment caused by VR headset movement. EEG data were recorded at a 1 kHz sampling rate using a BrainAmps Standard amplifier (Brain Products GmbH, Germany) during the experimental sessions. The Brain Vision Recorder software (Version 1.21.020121) managed the data acquisition process and monitored electrode connectivity. We utilized 64 active Ag/AgCl electrodes with integrated readout circuitry for noise cancellation and signal amplification. Electrodes were positioned according to the international 10–20 system, ensuring the Cz electrode was at the head’s vertex. The ground electrode was positioned at FPz, with the online reference at FCz. To maintain signal integrity, electrode-scalp impedance was kept below 10 k\(\Omega \). Data from channels T7, T8, FT7, and FT8 were excluded due to electromagnetic interference from the VR headset’s battery. Additionally, channels FT9, FT10, TP9, and TP10 were omitted because of inconsistent contact with the scalp. Figure 1a shows the setup of the experiment Experimental setup and protocol. (a) A participant wearing an EEG cap and a VR headset while resting their forearm on a towel for comfort. The participant had access to a keypad for self-assessment of their motor imagery performance. (b) A VR scene showing the virtual hand with the supernumerary thumb attached. The supernumerary thumb was modeled using Blender software (Version 3.422). (c) The experimental protocol included three main activities: a questionnaire, the main experiment with ten sessions, and a post-experiment survey. Each session involved embodiment exercise followed by sixteen trials, where participants mentally replicated movements involving flexion or extension of their natural thumb or the supernumerary thumb. Participants assessed their motor imagery execution after each trial. ST: Supernumerary thumb. The experiment consisted of three main activities:
• Kinesthetic visual imagery (KVI) questionnaire. The KVI questionnaire is a widely recognized tool for evaluating the visual and kinesthetic aspects of an individual’s motor imagery capabilities23. In this assessment, the experimenter performed and described five different movements, which are to be imagined by the participant in sequence. Each movement was to be imagined twice, once visually and the second kinesthetically. During evaluation, participants were assisted by the experimenter who helped them rate the clarity of their visual imagery and the vividness of kinesthetic sensations through a well-defined scale. Each modality, visual and kinesthetic, was scored out of a total of 25 points, distributed across the five movements. These are called as the visual MI score and the kinesthetic MI score.
• Main experiment. Participants were seated comfortably while experimenters set up the EEG equipment and fitted the VR headset (Meta Quest Pro24); see Fig. 1. The main experiment comprised 10 sessions, where the initial session was designated for practice. The session started with an embodiment period consisting of two 30-seconds phases of embodiment exercises. In the first phase, participants could see both hands and were instructed to pop balloons appearing in the virtual space using their right hand. The second phase introduced a virtual supernumerary thumb for participants to use in popping balloons attached to their right hand. Participants were directed to use their supernumerary thumb in popping balloons. This is to elicit a sense of embodiment of the virtual hand and the supernumerary thumb. After the embodiment period, participants underwent sixteen trials of the task. During these trials, they were instructed to rest their right forearm on a comfortable towel with their right palm facing them (see Fig. 1a). In the VR environment, participants could see their hand equipped with a supernumerary thumb, as shown in Fig. 1b. Each trial commenced with the hand in a static state for 2 seconds, followed by a motor observation period of another 2 seconds, during which a pre-recorded movement of either the natural or the supernumerary thumb performing flexion or extension was shown. The virtual hand then disappeared and reemerged in the corresponding pose. A baseline period followed for 2 seconds, displaying a fixation cross aligned with the center and just in front of the palm. After the cross vanished, participants were asked to engage in motor imagery of the previously observed movement, and during this time the the virtual hand and supernumerary thumb were static. Subsequently, participants were asked to rate their success in performing the motor imagery using a keypad with their left hand (binary rating). Consequently, each participant completed 160 trials, with the initial 16 trials conducted during the practice session. These trials were evenly divided among the four conditions: flexion and extension movements for both the natural and supernumerary thumbs. Figure 1c illustrates the experimental timeline, detailing the sequence of events for an individual trial.
• Post-experiment survey. After the completion of the 10 sessions, participants were asked to complete a questionnaire capturing their experience. Particularly, we asked participants to report on: (1) their ownership and agency of the virtual hand and (2) their ability to perform motor imagery for the natural and supernumerary thumbs. Those were assessed using a 5-point Likert scale. Pre-processing and analysis of the EEG data were performed offline using MATLAB release 2022a (MathWorks, United States) and EEGLAB toolbox (v14.1.2)25. EEG data were filtered using a passband Sinc FIR filter between 0.1 and 40 Hz. To mitigate artifacts from muscle activity, eye movements, and blinks, as well as to identify and exclude heavily affected channels, the Artifact Subspace Reconstruction (ASR)26 technique was used with the following settings: argflatline = 10, arghighpass = [0.025 0.075], argchannel = 0.8, argnoisy = 4, argburst = 20, argwindow = ‘off’. Applying ASR with a threshold (argburst) of 20 has been shown to effectively clean EEG data from ocular artifact while retaining the neural features of the data27. Subsequently, the Common Average Referencing (CAR)28 method was applied for re-referencing the EEG data, and the online reference channel, FCz, was retained. All trials that were rated as unsuccessful during the evaluation period were discarded. Approximately, 85% of the MI trials were retained, having been rated as successful, such that 86% and 84% of the natural and supernumerary thumb trials were retained, respectively. The two periods of focus in this study are the motor observation period and the motor imagery period. EEG data were epoched relative to the onset of each period. This epoch window spanned from 2 seconds before to 3 seconds after the onset. We implemented time-frequency analysis by convolving the signal with a set of complex Morlet wavelets, defined as complex sine waves tapered by a Gaussian. The frequencies of the wavelets ranged from 1 to 40 Hz in 40 linearly spaced 1 Hz steps. The full-width at half-maximum (FWHM) ranged from about 380 ms to 100 ms with increasing wavelet peak frequency. This corresponds to the wavelets with a number of cycles that increases logarithmically from 2 to 10. The power of the transformed signals is then calculated by squaring their amplitude. For trial-averaged analysis, epochs of the same condition were then averaged for each participant. For spectral power analysis, we calculated the event related spectral perturbation (ERSP) and applied decibel conversion to normalize the power of the ERSPs with respect to the baseline following the formula: \(10 \log _{10}\left( \frac{\text {activity}}{\textrm{baseline}}\right) \). Baseline normalization was applied to the data using a baseline window from \(-1.25\) to \(-0.25\) seconds29 relative to the onset for each period of interest. As for the ERD calculation of the alpha band, we followed the methodology outlined by Pfurtscheller30,31. Specifically, we first bandpass filtered the epoched trials from 9 to 13 Hz. Next, we squared the amplitude samples to obtain power samples and then averaged these power samples over trials. Finally, we averaged a small number of consecutive power samples (i.e., 50 datapoints corresponding to 50 ms) to reduce variance and smoothen the signal. ERD was then calculated using the formula \( \text {ERD} = 100 \times \frac{A – R}{R} \), where \( A \) is the activity during the epoch and \( R \) is the baseline.. Other frequency bands did not exhibit any neural patterns that are relevant to this study. We identified the ROIs for the trial-averaged statistical analysis to examine the desynchronization in the alpha band during the motor imagery and the motor observation periods using a data-driven approach. This approach is based on conditions averaging, rather than conditions difference, thus avoiding double-dipping32. For each of the MO and MI periods, both the ERSPs of both conditions (natural and supernumerary) were averaged together, and the mean alpha band power throughout the periods was compared to the mean of the baseline period. Regions were selected for further analysis if they exhibited a statistically significant reduction in alpha power relative to the baseline. Particularly, cortical regions that showed prominent discrepancy with respect to the baseline were identified. Activation within the identified regions was statistically compared to the baseline by pooling electrode data from each cortical region and performing statistical tests with respect to the baseline. The p-values are corrected with respect to the number of comparisons (i.e. identified regions). This selection method, grounded on averaged conditions, is orthogonal to condition differences thus avoiding circular inference32. For the motor imagery period, three ROIs were identified: 1) Frontal: Fp1, Fp2, AF3, AF4, AFz; 2) Contralateral sensorimotor: P1, P3, P5, CP1, CP3, CP5, C1, C3, C5; and 3) Ipsilateral sensorimotor: P2, P4, P6, CP2, CP4, CP6, C2, C4, C6. On the other hand, another set of ROIs were identified for the motor observation period: 1) Frontal: Fp1, Fp2, AF3, AF4, AFz; 2) Contralateral parieto-occipital: CP3, CP1, P7, P5, P3, P1, PO7, PO3, O1 and 3) Ipsilateral parieto-occipital: P4, P6, P8, PO4, PO8, O2. In accordance with our selection, the sensorimotor area has been associated with MI, while the parieto-occipital region has been reported to be involved during MO14,18. Figure 2 shows the ROIs selection based on the average-of-conditions for each period (i.e., MO, MI). Regions of interest. The region of interests were selected in a data-driven approach (MO: motor observation; MI: motor imagery; NT: natural thumb; ST: supernumerary thumb). The average power of all conditions was compared to the average baseline, defined as the period between \(-1.25\) and \(-0.25\) seconds before the onset. Highlighted regions are considered as ROIs as their power were significantly larger relative to the baseline (\(p < 0.05\), Wilcoxon sign-rank test). For trial-averaged analysis, we averaged the channels within each identified ROI. As per the aim of this study, we have conducted three main comparisons regarding the ERD in the identified ROIs: (1) between the motor imagery of the supernumerary thumb and rest; (2) between the natural and supernumerary thumbs during the motor imagery period; and (3) between flexion and extension movements of the supernumerary thumb throughout the motor imagery period. We further explored differences in the neural signatures between the natural and supernumerary thumbs during the motor observation period. All of these analyses were performed at the group level, with trials from individual participants averaged together. ERDs within the ROIs were segmented into 0.2 second time windows, each overlapping by 50%, and activation within each bin was averaged. To mitigate Type I error and adjust for multiple comparisons, the false discovery rate (FDR) method was applied to all p-values, such that for each analysis, a total of 60 comparisons (20 overlapping time-intervals \(\times \) 3 ROIs) were accounted for in each analysis. This manuscript reports corrected p-values and identifies the time intervals where statistically significant differences were found. Surrogate tests are ideal for EEG data analysis because they make no assumptions about data distribution; therefore, for each time window, we use the bootstrap surrogate test provided by the EEGLAB toolbox25,33. For the statistical analysis conducted to compare the improvement in classification accuracy after fine-tuning when performing single-trial classification, we used the paired t-test after confirming normality and corrected for multiple comparisons (5 comparisons,) using FDR. All statistical analyses were conducted using the Statistics toolbox in MATLAB 2022a and EEGLAB toolbox v.14.1.2. We perform four binary classification tasks. The first task utilizes the EEG trials to distinguish between the supernumerary thumb MI and the baseline. The second task attempts to classify the natural thumb and the supernumerary thumb during the motor imagery period. In the third task, we aim to classify between flexion and extension of the supernumerary thumb during MI. The fourth and final task focuses on distinguishing supernumerary thumb MO from natural thumb MO. The dataset consists of data from \(N=16\) subjects. Each subject’s data consisted of approximately \(tr=30\) trials per condition, with each condition defined as either flexion or extension of the natural or supernumerary thumbs (some trials were discarded due to unsuccessful motor imagery). The deep learning model takes as input a single trial \(X_i \in {\mathbb {R}}^{C\times T}\) where C is the number of EEG channels (electrodes) and T is the number of time points in a trial. The model aims to map the input trial \(X_i\) to its corresponding label \(y_i\). Thus, the dataset consists of \(S = \{X_i, y_i\}_{i=1}^m\), where \(C = 56\) (channels), \(T = 2000\) (data points; corresponding to a duration of 2 seconds with a sampling rate of 1 kHz), and m is the total number of trials. In this paper, we use ATCNet34, an attention-based temporal convolutional network architecture for the classification tasks. This model consists of three main blocks: a convolutional (CV) block, an attention (AT) block, and a temporal convolutional (TC) block. The hyperparameters of the model are the same as in34, except for the dropout rate of the CV block which is 0.1 and number of sliding windows after the CV layer which is 9 in our case. The model was evaluated using the leave-one-subject-out strategy in which the data from all the subjects except one was used for training the model and the left out subject was utilized to evaluate the performance of the model. Leave-one-subject-out ensures that the model is able to learn the underlying features in the EEG data corresponding to the task that are independent from the subjects. The model was trained for 30 epochs with a batch size of 16. Binary Cross Entropy (BCE) loss was chosen for the loss function since the tasks are binary classifications. Adam optimizer was utilized with a learning rate of 0.00005 and a weight decay of 0.01. The model training was performed on an NVIDIA Quadro RTX 5000 GPU. For each of the analyses presented in the paper, the mean and standard deviation of the accuracy across all 16 subjects were calculated.
The outcomes of the pre- and post-experiment questionnaires are analyzed to confirm the subjects’ ability to perform MI of the supernumerary thumb. Once confirmed, we conduct a step-by-step analysis of the EEG data. First, we examine if there exists a neural signature for the MI of the supernumerary thumb. If so, we determine whether this signature is distinct from that of the natural thumb. Next, we investigate the directional movement of the supernumerary thumb to see if there is a difference between the neural signatures of MI of flexion and MI of extension. Lastly, we analyze the neural activation during the motor observation period to identify any significant differences during passive observation of the movement of the supernumerary thumb versus the natural thumb. For each analysis, we employ trial-averaged statistical analysis and deep learning classification to validate the results on a group-level and single-trial basis, respectively. Before the experiment, participants were asked to complete the KVI Questionnaire23 to provide a baseline of their MI abilities. Participants showed a decent MI ability, scoring an average of 22.4 (SD = 2.5) out of 25 for visual MI and 18.4 (SD = 4.3) out of 25 for kinesthetic MI. After the experiment, participants were asked to complete a survey that evaluated their embodiment within the VR space, as well as their ability to perform MI for the natural thumb and the supernumerary thumb. On a scale of 5, participants reported a sense of ownership (M= 3.67, SD = 0.65) and agency (M = 3.95, SD = 0.69) towards their virtual hand. Participants rated highly their ability to perform MI for the natural thumb (M = 3.80, SD = 0.70) and MI for the supernumerary thumb (M = 3.90, SD = 0.64), with no significant differences between the two. The trial-averaged analysis shows that there exists a clear ERD pattern in the alpha band during the MI of the supernumerary thumb. Figure 3a shows the grand average topography map of the ERD during the motor imagery period, normalized with respect to the baseline. The figure shows a clear neural activation in the identified ROIs. The ERD in the three ROIs is significantly higher than their counterparts during the baseline period (i.e. zero ERD), as depicted in Fig. 3b. The ability to identify neural signatures of single-trial MI of the supernumerary thumb is confirmed through deep learning classification against baseline. We employed the ATCNet model34 for this classification task, as well as for the rest of the classification tasks in the remainder of this manuscript. Figure 3c shows the leave-one-subject-out classification accuracy, before and after fine-tuning. The average accuracy obtained is approximately 78.7% and 82.1%, for before and after fine-tuning, respectively, where 50% of the data of the test subject are used for fine-tuning, and only the remaining 50% were used in testing. MI of supernumerary thumb versus baseline. (a) The grand-averaged topography plot showing the alpha ERD during the MI of the supernumerary thumb. (b) Averaged alpha ERD compared to baseline for each ROI. \(p \) < 0.001(***), bootstrap test, FDR correction, (N = 16 samples). (c) Classification accuracy of MI of the supernumerary thumb versus baseline under leave-one-subject-out cross validation before and after fine-tuning with 50% of the test-subject data. The average accuracy is presented as a dashed blue (without fine-tuning) or red (with fine-tuning) line. ST: Supernumerary thumb. Natural thumb versus supernumerary thumb during the motor imagery period. (a) ERSP heatmaps for the three ROIs for the motor imagery period. (b) Time course of scalp topographies showing the alpha ERD with a resolution of 0.2 seconds averaged over all participants. (c) Time course of the grand-averaged alpha ERD for the three ROIs identified for the motor imagery period. The shaded area marks the time interval where a statistically significant difference in ERD was observed between the natural and supernumerary thumb motor imagery (\(p < 0.05\), bootstrap test, FDR correction); \(t_{range}\)= [0.5 to 0.9 seconds]). NT: Natural thumb, ST: Supernumerary thumb. Classification of the natural thumb versus supernumerary thumb during the motor imagery period. (a) Classification accuracy of MI of the supernumerary thumb versus the MI of the natural thumb under leave-one-subject-out cross validation before and after fine-tuning with 50% of the test-subject data. The average accuracy is presented as a dashed blue (without fine-tuning) or red (with fine-tuning) line. (b) The change in the average accuracy with the increase of the percentage of the data used from the test subject for fine-tuning. The statistical significance of the change is assessed; \(p<\) 0.05(*), paired t-test, FDR correction. We compared the neural signatures of the natural and supernumerary thumbs during MI, combining effector flexion or extension. Figures 4a and b show the ERSP heatmaps for the three identified ROIs and the topographic maps in the alpha band, respectively. Through the trial-averaged analysis, it can be observed, as hypothesized, that the ERD is centered in the alpha band in the contralateral and ipsilateral sensorimotor areas. Furthermore, a statistically significant difference is observed in the ipsilateral sensorimotor region, where the supernumerary thumb exhibited a more pronounced ERD between 0.5 and 0.9 seconds from the onset of MI, shown in Fig. 4c (\(p < 0.05\), bootstrap test, FDR corrected). From a single-trial perspective, detectable differences exist between MI activations of the natural and supernumerary thumbs. This is confirmed by the leave-one-subject-out classification results, shown in Fig. 5a. The average classification accuracy is 69.1% and 72.2% before and and after fine-tuning, respectively, where 50% of the data of the test subject are used for fine-tuning, and only the remaining 50% were used in testing. In Fig. 5b, a slight increase in the average classification accuracy is evident as more of the test-subject data is employed for fine-tuning. Notably, this increase in average accuracy becomes significantly greater than that of no fine-tuning once 50% of the test-subject data are utilized. Flexion versus extension movements for the supernumerary thumb during the motor imagery period. (a) ERSP heatmaps for the three ROIs for the motor imagery period. (b) Time course of scalp topographies showing alpha ERD with a resolution of 0.2 seconds averaged over all participants. (c) Time course of the grand-averaged alpha ERD for the three ROIs identified for the motor imagery period. The shaded area marks the time interval where a statistically significant difference in ERD was observed between the flexion and extension of the supernumerary thumb motor imagery (\(p < 0.05\), bootstrap test, FDR correction); \(t_{range}\)= [0.4–0.7 seconds and 0.8–1.2 seconds]). We further investigated differences in the neural signature between the MI of flexion and extension, motivated by the interest in directional control of the supernumerary thumb. Through the trial-averaged analysis, it can be observed that the MIs of both flexion and extension exhibit an ERD centered in the alpha band for the sensorimotor areas (Fig. 6a). Additionally, the MI corresponding to extension seems to introduce less localized ERD patterns across the scalp, particularly between 0.4 and 1.4 seconds from the onset; see Fig. 6b. Furthermore, Fig. 6c shows that, indeed, a statistically significant difference in the ERD between flexion and extension was found in the frontal region between 0.4 and 0.7 seconds and between 0.8 and 1.2 seconds from the MI onset (\(p < 0.05\), bootstrap test, FDR corrected). This can be visually detected in Fig.6a, where those differences appear to exist in lower frequency bands as well. The ability to differentiate between the MI of flexion from extension for the supernumerary thumb through a single-trial EEG data has also been attempted. The accuracy of the leave-one-subject-out classification is presented in Fig. 7a. The average classification accuracy is 60.1% and 65.8%, for before and after fine-tuning, respectively, where the impact of fine-tuning appears to be more pronounced for some subjects than for others. Figure 7b shows the increase in the average classification accuracy as more test subject data is used for fine-tuning. The average accuracy becomes significantly greater than the average accuracy without fine-tuning after utilizing at least 40% of the test-subject data. Classification of flexion versus extension movements for the supernumerary thumb during the motor imagery period. (a) Classification accuracy of MI of flexion versus extension for the supernumerary thumb under leave-one-subject-out cross validation before and after fine-tuning with 50% of the test-subject data. The average accuracy is presented as a dashed blue (without fine-tuning) or red (with fine-tuning) line. (b) The change in the average accuracy with the increase of the percentage of the data used from the test subject for fine-tuning. The statistical significance of the change is assessed; \(p \) < 0.05(*), paired t-test, FDR correction. Natural thumb versus supernumerary thumb during the motor observation period. (a) ERSP heatmaps for the three ROIs for the motor observation period. (b) Time course of scalp topographies showing alpha ERD with a resolution of 0.2 seconds averaged over all participants. (c) Time course of the grand-averaged alpha ERD for the three ROIs identified for the motor observation period. The shaded area marks the time interval where a statistically significant difference in alpha power was observed between the natural and supernumerary thumb motor observation (\(p < 0.05\), bootstrap test, FDR correction); \(t_{range}\)= [0.3–0.6 seconds]). NT: Natural thumb, ST: Supernumerary thumb. Classification of natural thumb versus supernumerary thumb during the motor observation period. (a) Classification accuracy of the supernumerary and the natural thumbs during MO under leave-one-subject-out cross validation before and after fine-tuning with 50% of the test-subject data. The average accuracy is presented as a dashed blue (without fine-tuning) or red (with fine-tuning) line. (b) The change in the average accuracy with the increase of the percentage of the data used from the test subject for fine-tuning. The statistical significance of the change is assessed; \(p<\) 0.05(*), paired t-test, FDR correction. As mentioned above, the motor observation period was incorporated into the experimental paradigm to guide and improve MI quality for the supernumerary thumb. A difference in alpha ERD between the natural thumb and the supernumerary thumb during MO indicates that the visual processing of both effectors is distinct at the neural level. Upon examining the ERD across the three ROIs during MO (see Fig. 2) and comparing the natural and supernumerary thumbs, we observed a more pronounced ERD in the frontal region for the supernumerary thumb. Figure 8a shows the ERSP heatmaps of the three ROIs, showing a clear and continuous desynchronization pattern during the motor observation period. Figure 8b depicts the time-course scalp topographies for the alpha ERD of both the natural and supernumerary thumbs during the motor observation period. The ERD of the alpha band at the frontal region is found to be significantly lower for the supernumerary thumb between 0.3 and 0.6 seconds from the MO onset, compared to the natural thumb (\(p<\) 0.05, bootstrap test, FDR corrected), as shown in Fig. 8c. This difference in the frontal region is visible from the time-course of scalp topographies (third time bin) shown in Fig. 8b. At the single-trial level, the deep learning model was able to classify the MO of the two thumbs apart with an average accuracy of 71.8% and 76.1%, before and after fine-tuning, respectively; see Fig. 9a. The average accuracy becomes significantly greater than the average accuracy without fine-tuning after utilizing at least 30% of the test-subject data (Fig. 9b).
Through this study, we aimed to evaluate the validity of VR as an environment for eliciting neural synergies associated with a supernumerary thumb and exploring its potential for MI training, motivated by the benefits VR offers in terms of ease of use, availability, and reduced MI training cost. The results showed that individuals in the VR environment were not only able to reliably perform MI for the supernumerary thumb during the MI period, but also able to do so in a manner that could distinguish the supernumerary thumb from the natural one. The single-trial classification results are also promising, indicating that although there may be a certain degree of personal features involved in the MI process, there may also be a degree of shared features that could be further explored. During the MI period, the trial-averaged difference between the supernumerary and natural thumbs was observed in the ipsilateral sensorimotor ROI, where the supernumerary thumb exhibited a statistically more pronounced bilateral ERD activation, compared to predominantly contralateral ERD activation for the natural thumb. Bilateral activation during unilateral MI is observed with movements of increased complexity35; a movement performed with a supernumerary effector is inherently more complex, and this may explain this difference. Another point of interest is the more widespread distribution of the ERD pattern across the scalp for the supernumerary thumb, consistent with findings from previous studies11. The MI of a supernumerary hand in VR has been shown to generate a widespread ERD in the alpha band across the scalp36, and previous studies have demonstrated that MI of an unfamiliar action recruits more brain resources and induces a stronger response than simpler or familiar actions37. Another hypothesis for this widespread ERD pattern is that the supernumerary thumb might be perceived as a tool rather than an owned body part. It has been reported that executing a task with a tool activates other brain regions than the motor cortex, such as the ipsilateral parietal region, more extensively than performing the same task with the hand38,39,40. The question of whether the extensive ERD pattern results from the supernumerary thumb being perceived as a tool, or due to the unfamiliarity of the effector, remains open and requires further investigation. A distinction was also found in the neural signatures associated with the MI of flexion and extension of the supernumerary thumb. Adding a directional control to a supernumerary effector is pivotal, as it allows for more complex movements and better usage of the effector. Despite the relatively low classification accuracy at the single-trial level, the observed neural distinction between flexion and extension of the supernumerary thumb hints at the potential for developing more discernible neural activations through longitudinal training. Interestingly, this difference was not observed during the MI of the natural thumb. We have also investigated the neural differences between the supernumerary and the natural thumbs during the MO period. The single-trial classification results (Fig. 9a) show a clear distinction between the two signatures. Furthermore, a significant difference in the alpha ERD is observed in the frontal region at the trial-averaged level. This indicates that the VR-based MO period showed a differentiation in the neural signature during the passive observation of the movement of the natural versus the supernumerary effector, possibly leading to a distinction between the two in the subsequent motor imagery period. This finding is also neural evidence of previous literature, reporting that providing realistic visual feedback that is designed to induce a sense of embodiment41,42, or a VR embodiment, assists in MI training43,44,45. Others reported the effectiveness of MO in promoting early learning of new and complex tasks46. Despite both MO and MI periods eliciting a clear ERD, each had its own associated region: MO activated the parieto-occipital area, while MI activated the sensorimotor cortex. In a previous study by Liu et al.11, subjects were instructed to perform MI of bending the supernumerary thumb while simultaneously engaging in MO of the action displayed on a screen. Trials that were labeled as low performance in regards to MI were found to exhibit a greater ERD activation in the parieto-occipital region. Our results explain this observation, highlighting that in those trials, subjects could have just performed MO disregarding MI. This finding provides evidence for the need to separate motor stimulation tasks (MO and MI) for the quality of EEG data, particularly in the context of BCI and deep learning classification. Furthermore, despite observing a desynchronization in the beta band during both MI and MO for the different conditions as can be seen from the ERSP heatmaps, no statistically significant differences were observed across the analyzed conditions. This could be explained by the different functional roles that each of the frequency bands play during MI and MO47. Our results also show the existence of neural activity in the frontal brain region. This is known to be associated with higher-order cognitive functions such as abstraction, and attention48. We hypothesize the activation of the frontal region is due to the two levels of abstraction involved. The first is caused by the VR environment where an ERD was observed for both, the natural and supernumerary thumb in the frontal region. The second layer of abstraction is for the supernumerary thumb being recognized as an artificial effector; the ERD in the frontal ROI was more pronounced for the supernumerary thumb during MO. This observation aligns with the findings by Penaloza et al.9 and Liu et al.11, where they reported that MI of a supernumerary effector (arm and thumb, respectively) in physical space influenced not just the sensorimotor areas but also the frontal area. The frontal ROI is a region that is susceptible to EOG artifacts. EOG artifacts generally appear in the delta band (extending to the theta band as well) as an increase in power in those bands with respect to non-contaminated EEG. However, the observed effects in our manuscript are desynchronizations in the alpha band, indicating a decrease in power, which is not characteristic of EOG artifacts49. The trial-averaged statistical analysis presented in this study was complemented by single-trial deep learning classification using the leave-one-subject-out cross-validation method. The average classification accuracies obtained significantly surpass the chance level, yet with notable variability across subjects, suggesting diverse capabilities in performing MI for a supernumerary thumb. These findings demonstrate the presence of generalized MI features for the supernumerary thumb across the population, as evidenced by the model’s ability to generalize to unseen test-subject data. Additionally, fine-tuning the model on a subset of test-subject data was observed to enhance average classification accuracy. The slight increase in classification accuracy observed for most subjects after fine-tuning could be due to merely including more data for training, but possibly due to the presence of personalized neural features. Further validation with additional control experiments are required to conclusively attribute the improvements to personalized features. It is also important to acknowledge the cross-sectional nature of this study. The study aims to investigate whether distinct MI neural signatures for a supernumerary effector can be elicited in a VR space using a novel paradigm involving sequential motor tasks, with the motivation of MI training. The results confirm that this is indeed possible on a cross-sectional level, meaning that the overall protocol was able to elicit these neural activations from participants who had no prior experience with the supernumerary effector. In this sense, participants seemed to have learned a new set of movements, those associated with the supernumerary thumb, and possibly could have improved in MI performance of these skills. However, given this is not a longitudinal study, it is difficult to make conclusions about the effectiveness of the paradigm in MI training, as well as the individual or combined effects of the MO and MI tasks. A longitudinal study would allow further understanding and evaluation of the paradigm and its components from an MI training perspective. Similarly, the results confirm that VR is a valid medium through which the distinct neural signature of a supernumerary effector can be elicited, but do not evaluate its performance against the physical space and the presence of a physical supernumerary effector. The goal of the study was to investigate the feasibility of using VR with the proposed paradigm, given the aforementioned considerations and other possible benefits. For example, VR could enable the customization of the virtual supernumerary effector without the need to construct a new device over every iteration and, in the case of a larger or delicate effector, could ensure the safety of both the individual and the device. Thus, VR could be low-cost alternative for the development and training environment for a supernumerary effector. However, given the disparity between the virtual and physical space, further research would be needed to understand and evaluate the benefits that VR can offer, as well as how additional modalities such as haptic feedback could help bridge the gap and enhance the quality of MI50,51,52. Collectively, this paper demonstrates the potential of VR as a platform for eliciting distinct MI signatures for supernumerary effectors, such as a supernumerary thumb, supported by single-trial and trial-averaged evidence. Future studies will focus on exploring the transition from VR to physical space following longitudinal training in VR, as well as investigating the effects of the proposed MI paradigm and its components in MI training. Furthermore, another interesting endeavor is to investigate whether, following longitudinal training, the activation associated with the supernumerary effector’s MI localizes, and whether this localization persists, thereby demonstrating either a short-term plasticity or a lasting adaptation.…Read more by Alsuradi, UAE, Computer Engineering, Department of Electrical, New York University, Sarmadi, Volcic, Division of Science, Health, Alireza, Salam, Mohamad, New York University Abu Dhabi, Center for Artificial Intelligence, S. Farokh, Atashzar, USA, Engineering Division, Farshad, Joseph, Abu Dhabi, Robert, New York, Center for Brain, Khorrami, Hanan, Eid, Hong, Haneen, Robotics
