BRAIN-BODY SENSORY FUSION: MERGING FNIRS/EEG NEUROIMAGING AND FULL-BODY MOTION CAPTURE

Functional connectivity between the brain and body kinematics has not been largely investigated due to the requirement of being motionless in neuroimaging techniques such as functional magnetic resonance imaging (fMRI). The importance of investigating this connectivity arises from the fact that the connectivity is disrupted in many neurodegenerative disorders such as Parkinson’s Disease (PD). PD is a neurological progressive disorder characterized by movement symptoms including slowness of movement, stiffness, tremors at rest, and walking and standing instability.\nBody kinematics are generally divided into two categories of fine and gross motor tasks. Fine motor movements are referred to small range kinematics of the body such as finger movements. Gross motor movements are related to bigger range movements such as limb motion. Unified Parkinson’s Disease Rating Scale (UPDRS) is a protocol consisting of 42 tasks that the patients need to perform for the neurologists to make a decision regarding the diagnosis or progression of the disease. 14 of these tasks are motor exams including both fine and gross movements. In this work, a smart wearable glove has been developed in order to measure fine motor movements of fingers. The performance of the glove regarding measuring finger movements has been tested and reported. The smart glove is tested on a robotic hand and healthy humans performing finger tapping task. The variability in human subjects compared to the robotic hand is clear based on the results of the test. The variance of finger tapping frequency on robotic hand is less than 6:00 x e-5compared to the variance for human participants which is in the range of 0:001 -0:1. A non-optical motion capture system (Mocap) consisting of inertia measurement unit (IMU) sensors has been utilized to record the gross movements of the body.\nDeveloping a framework for appropriately collecting simultaneous data from functional near infrared spectroscopy (fNIRS) neuroimaging system and motion capture system is discussed as the first aim. An application protocol interface (API) is developed in order to call each system’s software and record the data on separate files in milliseconds accuracy. The performance of this framework is tested at the system level to validate the data and also tested in real life experiment. A synchrony test is performed in order to validate if the recorded data from these systems through the API is synchronized. The results show that the data is acceptably synchronized with a time difference of few milliseconds which is negligible in fNIRS studies. The real life experiment of performing some gross motor tasks by healthy participants revealed that the brain activity and body movements detected through these systems are coupled. The promising results suggest that this framework can be developed for more complicated processing and experiments.\nThe second aim of this study is to develop algorithms in order to fuse and process the data recorded from the brain and body through the developed framework, and also developing a visualization user interface to monitor the brain activity and body kinematics simultaneously. Algorithms are developed in order to synchronize the recorded data, compensate the uneven sampling rates, calculate oxygenated hemoglobin levels in the brain from fNIRS, calculate the acceleration vector from Mocap, and normalize the flex sensor data from the smart glove.\nValidating the developed interface and algorithms in a human study is followed as the third aim of this study. 11 PD patients and 10 neurotypical (NT) healthy older participants were recruited in order to perform 4 of the UPDRS motor tasks including both fine and gross movements. The tasks include finger tapping, hand flipping, arm movement, and foot stomping on both left and right side limbs. Each task was explained for the participants and the participants were asked to practice each task. There was no recording during the practice time. Montreal cognitive assessment test and a questionnaire regarding the history of the disease is given to PD participants for further analysis by the neurologists. Body kinematics were recorded by the Mocap and smart glove. fNIRS and electroencephalogram (EEG) have been utilized as the brain neuroimaging systems. The hypothesis was: "There are differences between older healthy adults and PD patients which could be detected and distinguished by the fusion of brain activation and body movements"\nThe recorded data from each of the modalities have been analyzed individually, and the processed data has been used for classification between PD and NT group. The average changes of oxygenated hemoglobin from fNIRS, power spectral density of EEG in the Theta, Alpha, and Beta bands, acceleration vector from Mocap, and normalized flex sensor data were used for classification. 12 different support vector machine (SVM) classifiers have been used on different datasets such as only fNIRS data, only EEG data, hybrid fNIRS/EEG data, and all the fused data for two scenarios of classifying PD and NT based on each activity individually, and all the fused data together. The PD and NT group could be distinguished by the accuracy of more than 85% for each individual activity. For all the fused data, the accuracy of classification for differentiating PD and NT groups are 81:23%, 92:79%, 92:27%, and 93:40% for the fNIRS only, EEG only, hybrid fNIRS/EEG, and all the fused data, respectively. The results show that adding the data from each modality in the classification increases the accuracy, which implies that each modality carries useful information that results in a higher rate of distinguishing PD and NT. The lower accuracy of the fNIRS data only classification is due to the delay of the hemodynamic response which is variable for each individual subject. This makes it hard for the classifier to distinguish the two groups.\nThe promising results show the feasibility of using this brain-body fusion system to distinguish the differences in PD and NT group regarding the fused information of brain and body. The future goal is to develop an interface to provide the visualization of brain activity and body kinematics of the participant simultaneously while the participant is performing the motor tasks. This is happening by a visual observance of the patients by neurologists. The visualization interface can provide a tool for neurologists to observe both body movements and brain activity simultaneously. Also, adding the results of the analysis will be provided at the end of the experiment which can be used as a supporting tool for neurologists to make a decision about diagnosis or progression of PD. Considering more individual information from the patients such as their UPDRS scores, type of medication, or dosage of medication in the machine learning analysis can result in a multi-class classification capable of distinguishing each patient individually. This approach can be helpful in monitoring the progress of the disease or considering the effectiveness of the medication. This needs a quantitative evaluation of sample size calculation and longitudinal data collection in order to generalize the results of this dissertation.

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.