- Rehabilitation robotics, Human-Robot Interaction, Rehabilitation, Psychophysiology, Pattern Recognition, Machine Learning, and 5 moreRobotics, Virtual Reality (Computer Graphics), Affective Computing, Haptics, and Physiological Computing(Robotics, Virtual Reality (Computer Graphics), Affective Computing, Haptics, and Physiological Computing)edit
- Vesna Novak (formerly Domen Novak) is an Associate Professor in the Department of Electrical Engineering and Computer... moreVesna Novak (formerly Domen Novak) is an Associate Professor in the Department of Electrical Engineering and Computer Science at the University of Cincinnati. She received her diploma and PhD in electrical engineering from the University of Ljubljana in 2008 and 2011, respectively. She was a postdoctoral fellow in the Sensory-Motor Systems Lab at ETH Zurich, Switzerland, from 2012 to 2014, and then an Assistant and Associate Professor at the University of Wyoming from 2014 to 2021. Her research interests include rehabilitation robotics, wearable robotics, virtual reality, serious games, affective computing, and human activity recognition.(Vesna Novak (formerly Domen Novak) is an Associate Professor in the Department of Electrical Engineering and Computer Science at the University of Cincinnati. She received her diploma and PhD in electrical engineering from the University of Ljubljana in 2008 and 2011, respectively. She was a postdoctoral fellow in the Sensory-Motor Systems Lab at ETH Zurich, Switzerland, from 2012 to 2014, and then an Assistant and Associate Professor at the University of Wyoming from 2014 to 2021. Her research interests include rehabilitation robotics, wearable robotics, virtual reality, serious games, affective computing, and human activity recognition.)edit
Rehabilitation after neurological injury can be provided by robots that help patients perform different exercises. Multiple such robots can be combined in a rehabilitation robot gym to allow multiple patients to perform a diverse range of... more
Rehabilitation after neurological injury can be provided by robots that help patients perform different exercises. Multiple such robots can be combined in a rehabilitation robot gym to allow multiple patients to perform a diverse range of exercises simultaneously. In pursuit of better multipatient supervision, we aim to develop an automated assignment system that assigns patients to different robots during a training session to maximize their skill development. Our previous work was designed for simplified simulated environments where each patient's skill development is known beforehand. The current work improves upon that work by changing the deterministic environment into a stochastic environment where part of the skill development is random and the assignment system must estimate each patient's predicted skill development using a neural network based on the patient's previous training success rate with that robot. These skill development estimates are used to create patient-robot assignments on a timestep-by-timestep basis to maximize the skill development of the patient group. Results from simplified simulation trials show that the schedules produced by our assignment system outperform multiple baseline schedules (e.g., schedules where patients never switch robots and schedules where patients only switch robots once halfway through the session). Additionally, we discuss how some of our simplifications could be addressed in the future.
Research Interests: Robotics, Stochastic Process, Artificial Intelligence, Human Computer Interaction, Machine Learning, and 15 moreSerious Games, Motor Learning, Musculoskeletal Rehabilitation, Scheduling, Optimization techniques, Neural Networks, Modeling and Simulation, Human-Robot Interaction, Stroke rehabilitation, Physical Medicine and Rehabilitation, Rehabilitation robotics, Artificial Neural Networks, Scheduling Algorithms, Task Allocation, and Task Scheduling(Serious Games, Motor Learning, Musculoskeletal Rehabilitation, Scheduling, Optimization techniques, Neural Networks, Modeling and Simulation, Human-Robot Interaction, Stroke rehabilitation, Physical Medicine and Rehabilitation, Rehabilitation robotics, Artificial Neural Networks, Scheduling Algorithms, Task Allocation, and Task Scheduling)
(Serious Games, Motor Learning, Musculoskeletal Rehabilitation, Scheduling, Optimization techniques, Neural Networks, Modeling and Simulation, Human-Robot Interaction, Stroke rehabilitation, Physical Medicine and Rehabilitation, Rehabilitation robotics, Artificial Neural Networks, Scheduling Algorithms, Task Allocation, and Task Scheduling)
Passive back support exoskeletons, which support the human trunk using elements like springs and elastic bands, have demonstrated positive results in laboratory-based studies, but have seen significantly less field testing. As an... more
Passive back support exoskeletons, which support the human trunk using elements like springs and elastic bands, have demonstrated positive results in laboratory-based studies, but have seen significantly less field testing. As an intermediate step between generic lab evaluations and field tests, we conducted a single-session lab evaluation of the HeroWear Apex exoskeleton with mockup construction tasks: 20 adult men (without extensive construction experience) lifted, carried and raised lumber boards (265 cm length, up to 18 kg total load). The exoskeleton significantly reduced mean erector spinae electromyograms, with effect sizes (Cohen’s d) ranging from -0.2 to -0.55 – corresponding to reductions of 5-25% relative to no-exoskeleton electromyogram values. In asymmetric carrying tasks, the exoskeleton provided more assistance to the more heavily loaded erector spinae muscle. Additionally, in lifting tasks, the exoskeleton decreased trunk/hip flexion/extension range of motion and increased knee range of motion, indicating changes in lifting strategy. These results indicate potential exoskeleton benefits for lumber board carrying and will serve as the basis for further evaluations with workers in the field.
Research Interests:
Low-cost exoskeletons can effectively support workers in physically demanding jobs, but most such exoskeletons have been developed to support repetitive lifting or uncomfortable static postures. Very few low-cost exoskeletons have been... more
Low-cost exoskeletons can effectively support workers in physically demanding jobs, but most such exoskeletons have been developed to support repetitive lifting or uncomfortable static postures. Very few low-cost exoskeletons have been designed to support walking while carrying heavy objects, which would be beneficial for jobs such as moving furniture and warehouse work. This paper thus presents a single-session lab evaluation of the Auxivo CarrySuit, a low-cost upperbody exoskeleton designed for carrying objects that would normally be held with the arms. Twenty participants carried four loads (box or two bags, 20 or 40 lb total weight) for 2 minutes each on a treadmill with and without the CarrySuit. Across all loads, the CarrySuit significantly reduced the mean electromyogram of the middle trapezius (partial eta-squared = .74from 16.1% to 8.8% of maximum voluntary contraction value) and anterior deltoid (partial eta-squared = .26from 3.0% to 1.1% of maximum voluntary contraction value) with no corresponding increase in lower back muscle activation. Furthermore, maximum heart rate and Ratings of Perceived Exertion were also reduced by the CarrySuit, and discomfort was shifted from the upper body to the legs. While arm EMG was not measured, it is likely that it was also reduced due to the unloading of the arms. The CarrySuit can thus be considered beneficial in the short term, though longer-term evaluations with actual workers are needed to determine practical benefits.
Research Interests: Robotics, Biomedical Engineering, Ergonomics, Biomechanics, Human Factors, and 12 moreGait Biomechanics, Occupational Health & Safety, Exoskeletons, Assistive Technology, Wearable Technologies, Orthotics, Prosthetics and orthotics, Wearable Technology, Electromyography, Gait and Posture, Wearable Robots, and Assistive Technologies
Two people's physiological responses become more similar as those people talk or cooperate, a phenomenon called physiological synchrony. The degree of synchrony correlates with conversation engagement and cooperation quality, and could... more
Two people's physiological responses become more similar as those people talk or cooperate, a phenomenon called physiological synchrony. The degree of synchrony correlates with conversation engagement and cooperation quality, and could thus be used to characterize interpersonal interaction. In this study, we used a combination of physiological synchrony metrics and pattern recognition algorithms to automatically classify four different dyadic conversation scenarios: two-sided positive conversation, two-sided negative conversation, and two one-sided scenarios. Heart rate, skin conductance, respiration and peripheral skin temperature were measured from 16 dyads in all four scenarios, and individual as well as synchrony features were extracted from them. A two-stage classifier based on stepwise feature selection and linear discriminant analysis achieved a four-class classification accuracy of 75.0% in leave-dyad-out crossvalidation. Removing synchrony features reduced accuracy to 65.6%, indicating that synchrony is informative. In the future, such classification algorithms may be used to, e.g., provide realtime feedback about conversation mood to participants, with applications in areas such as mental health counseling and education. The approach may also generalize to group scenarios and adjacent areas such as cooperation and competition.
Research Interests: Physiology, Human Physiology, Interpersonal Communication, Psychophysiology, Machine Learning, and 12 moreClassification (Machine Learning), Pattern Recognition, Clustering and Classification Methods, Affective Neuroscience, Affective Computing, Biomedical signal and image processing, Affect/Emotion, Affect, Physiological Computing, Biomedical Signal Processing, Physiological Psychology, and Dyads
Background: A robotic rehabilitation gym can be defined as multiple patients training with multiple robots or passive sensorized devices in a group setting. Recent work with such gyms has shown positive rehabilitation outcomes;... more
Background: A robotic rehabilitation gym can be defined as multiple patients training with multiple robots or passive sensorized devices in a group setting. Recent work with such gyms has shown positive rehabilitation outcomes; furthermore, such gyms allow a single therapist to supervise more than one patient, increasing cost-effectiveness. To allow more effective multipatient supervision in future robotic rehabilitation gyms, we propose an automated system that could dynamically assign patients to different robots within a session in order to optimize rehabilitation outcome. Methods: As a first step toward implementing a practical patient-robot assignment system, we present a simplified mathematical model of a robotic rehabilitation gym. Mixed-integer nonlinear programming algorithms are used to find effective assignment and training solutions for multiple evaluation scenarios involving different numbers of patients and robots (5 patients and 5 robots, 6 patients and 5 robots, 5 patients and 7 robots), different training durations (7 or 12 time steps) and different complexity levels (whether different patients have different skill acquisition curves, whether robots have exit times associated with them). In all cases, the goal is to maximize total skill gain across all patients and skills within a session. Results: Analyses of variance across different scenarios show that disjunctive and time-indexed optimization models significantly outperform two baseline schedules: staying on one robot throughout a session and switching robots halfway through a session. The disjunctive model results in higher skill gain than the time-indexed model in the given scenarios, and the optimization duration increases as the number of patients, robots and time steps increases. Additionally, we discuss how different model simplifications (e.g., perfectly known and predictable patient skill level) could be addressed in the future and how such software may eventually be used in practice. Conclusions: Though it involves unrealistically simple scenarios, our study shows that intelligently moving patients between different rehabilitation robots can improve overall skill acquisition in a multi-patient multi-robot environment. While robotic rehabilitation gyms are not yet commonplace in clinical practice, prototypes of them already exist, and our study presents a way to use intelligent decision support to potentially enable more efficient delivery of technologically aided rehabilitation.
Research Interests: Robotics, Human Computer Interaction, Multiagent Robotics, Rehabilitation, Mixed-integer Linear Programming, and 10 moreHaptics, Optimization techniques, Human-Robot Interaction, Physical Medicine and Rehabilitation, Rehabilitation robotics, Physical Therapy, Stroke, Multi Robot Systems, Human Robot Interaction, and Task Scheduling
A robotic rehabilitation gym is a setup that allows multiple patients to exercise together using multiple robots. The effectiveness of training in such a group setting could be increased by dynamically assigning patients to specific... more
A robotic rehabilitation gym is a setup that allows multiple patients to exercise together using multiple robots. The effectiveness of training in such a group setting could be increased by dynamically assigning patients to specific robots. In this simulation study, we develop an automated system that dynamically makes patient-robot assignments based on measured patient performance to achieve optimal group rehabilitation outcome. To solve the dynamic assignment problem, we propose an approach that uses a neural network classifier to predict the assignment priority between two patients for a specific robot given their task success rate on that robot. The priority classifier is trained using assignment demonstrations provided by a domain expert. In the absence of real human data from a robotic gym, we develop a robotic gym simulator and create a synthetic dataset for training the classifier. The simulation results show that our approach makes effective assignments that yield comparable patient training outcomes to those obtained by the domain expert.
Research Interests: Robotics, Robotics (Computer Science), Rehabilitation, Cognitive Robotics, Human-Robot Interaction, and 8 moreStroke rehabilitation, Rehabilitation robotics, Physical Therapy, Motor Control and Rehabilitation (Health Sciences), Mechatronics and Robotics, Human Robot Interaction, Physical Human-Robot Interaction, and Task Scheduling
Transgender and gender diverse people often experience voice-gender incongruence, which is inversely correlated with health and quality of life. Such incongruence could be reduced with voice and communication training, but... more
Transgender and gender diverse people often experience voice-gender incongruence, which is inversely correlated with health and quality of life. Such incongruence could be reduced with voice and communication training, but expert-administered training is often inaccessible while self-guided training is difficult and potentially risky. Training could alternatively be provided through software (eg, smartphone apps), but such software is at an early stage. This qualitatively driven mixed-methods study thus includes surveys and interviews with 21 transfeminine, transmasculine and nonbinary people to identify general views of voice and communication training software as well as most desirable features of such software. Participants were positive about the general idea and described ways to effectively implement four critical features: feedback, accountability, automated goal setting, and training characteristics other than pitch. They also discussed optional or undesirable features. These findings may inform development of voice and communication training software, thus improving health and quality of life for gender minorities.
Research Interests: Human Computer Interaction, Educational Technology, Serious Games, Transgender Studies, Human-Computer Interaction for Games, and 15 moreTransgender Health, Transgender, Games for Learning, Voice, Mobile apps, Transsexuality and Transgender, Gender Diversity, Mobile Human Computer Interaction, LGBTQ studies, Non-Binary Gender, Transgender Voice, Smartphone apps, Nonbinary Gender Idenitity, Voice Training, and Voice Disorders; Transgender Voice and Communication(Transgender Health, Transgender, Games for Learning, Voice, Mobile apps, Transsexuality and Transgender, Gender Diversity, Mobile Human Computer Interaction, LGBTQ studies, Non-Binary Gender, Transgender Voice, Smartphone apps, Nonbinary Gender Idenitity, Voice Training, and Voice Disorders; Transgender Voice and Communication)
(Transgender Health, Transgender, Games for Learning, Voice, Mobile apps, Transsexuality and Transgender, Gender Diversity, Mobile Human Computer Interaction, LGBTQ studies, Non-Binary Gender, Transgender Voice, Smartphone apps, Nonbinary Gender Idenitity, Voice Training, and Voice Disorders; Transgender Voice and Communication)
Back support exosuits can support workers in physically demanding jobs by reducing muscle load, which could reduce risk of work-related musculoskeletal disorders. This paper presents a two-session evaluation of a commercial exosuit, the... more
Back support exosuits can support workers in physically demanding jobs by reducing muscle load, which could reduce risk of work-related musculoskeletal disorders. This paper presents a two-session evaluation of a commercial exosuit, the Auxivo LiftSuit 1.1. In session 1, 17 participants performed single repetitions of lifting and static leaning tasks with and without the LiftSuit. In session 2, 10 participants performed 50 box lifting repetitions with and without the LiftSuit. In session 1, the exosuit was considered mildly to moderately helpful, and reduced erector spinae and middle trapezius electromyograms. In session 2, the exosuit was not considered helpful, but reduced the middle trapezius electromyogram and trunk and thigh ranges of motion. These effects are likely due to placement of elastic elements and excessive stiffness at the hips. Overall, the LiftSuit appears suboptimal for long-term use, though elastic elements on the upper back may reduce muscle activation in future exosuit designs.
Research Interests:
Physiological responses of two interacting individuals contain a wealth of information about the dyad: for example, the degree of engagement or trust. However, nearly all studies on dyadic physiological responses have targeted group-level... more
Physiological responses of two interacting individuals contain a wealth of information about the dyad: for example, the degree of engagement or trust. However, nearly all studies on dyadic physiological responses have targeted group-level analysis: e.g., correlating physiology and engagement in a large sample. Conversely, this paper presents a study where physiological measurements are combined with machine learning algorithms to dynamically estimate the engagement of individual dyads. Sixteen dyads completed 15-min naturalistic conversations and self-reported their engagement on a visual analog scale every 60 s. Four physiological signals (electrocardiography, skin conductance, respiration, skin temperature) were recorded, and both individual physiological features (e.g., each participant’s heart rate) and synchrony features (indicating degree of physiological similarity between two participants) were extracted. Multiple regression algorithms were used to estimate self-reported engagement based on physiological features using either leave-interval-out crossvalidation (training on 14 60-s intervals from a dyad and testing on the 15th interval from the same dyad) or leave-dyad-out crossvalidation (training on 15 dyads and testing on the 16th). In leave-interval-out crossvalidation, the regression algorithms achieved accuracy similar to a ‘baseline’ estimator that simply took the median engagement of the other 14 intervals. In leave-dyad-out crossvalidation, machine learning achieved a slightly higher accuracy than the baseline estimator and higher accuracy than an independent human observer. Secondary analyses showed that removing synchrony features and personality characteristics from the input dataset negatively impacted estimation accuracy and that engagement estimation error was correlated with personality traits. Results demonstrate the feasibility of dynamically estimating interpersonal engagement during naturalistic conversation using physiological measurements, which has potential applications in both conversation monitoring and conversation enhancement. However, as many of our estimation errors are difficult to contextualize, further work is needed to determine acceptable estimation accuracies.
Research Interests:
Trunk exoskeletons are wearable devices that support humans during physically demanding tasks by reducing biomechanical loads on the back. While most trunk exoskeletons are rigid devices, more lightweight soft exoskeletons (exosuits) have... more
Trunk exoskeletons are wearable devices that support humans during physically demanding tasks by reducing biomechanical loads on the back. While most trunk exoskeletons are rigid devices, more lightweight soft exoskeletons (exosuits) have recently been developed. One such exosuit is the HeroWear Apex, which achieved promising results in the developers’ own work but has not been independently evaluated. This paper thus presents an evaluation of the Apex with 20 adult participants during multiple brief tasks: standing up from a stool with a symmetric or asymmetric load, lifting a unilateral or bilateral load from the floor to waist level, lifting the same bilateral load with a 90-degree turn to the right, lowering a bilateral load from waist level to floor, and walking while carrying a bilateral load. The tasks were performed in an ABA-style protocol: first with exosuit assistance disengaged, then with it engaged, then disengaged again. Four measurement types were taken: electromyography (of the erector spinae, rectus abdominis, and middle trapezius), trunk kinematics, self-report ratings, and heart rate. The exosuit decreased the erector spinae electromyogram by about 15% during object lifting and lowering tasks; furthermore, participants found the exosuit mildly to moderately helpful. No adverse effects on other muscles or during non-lifting tasks were noted, and a decrease in middle trapezius electromyogram was observed for one task. This confirms that the HeroWear Apex could reduce muscle demand and fatigue. The results may transfer to other exoskeletons with similar design principles, and may inform researchers working with other wearable devices.
Research Interests:
Lifting and carrying heavy objects is a major aspect of physically intensive jobs. Wearable sensors have previously been used to classify different ways of picking up an object, but have seen only limited use for automatic classification... more
Lifting and carrying heavy objects is a major aspect of physically intensive jobs. Wearable sensors have previously been used to classify different ways of picking up an object, but have seen only limited use for automatic classification of load position and weight while a person is walking and carrying an object. In this proof-of-concept study, we thus used wearable inertial and electromyographic sensors for offline classification of different load positions (frontal vs. unilateral vs. bilateral side loads) and weights during gait. Ten participants performed 19 different carrying trials each while wearing the sensors, and data from these trials were used to train and evaluate classification algorithms based on supervised machine learning. The algorithms differentiated between frontal and other loads (side/none) with an accuracy of 100%, between frontal vs. unilateral side load vs. bilateral side load with an accuracy of 96.1%, and between different load asymmetry levels with accura...
Research Interests:
In competitive and cooperative scenarios, task difficulty should be dynamically adapted to suit people with different abilities. State-of-the-art difficulty adaptation methods for such scenarios are based on task performance, which... more
In competitive and cooperative scenarios, task difficulty should be dynamically adapted to suit people with different abilities. State-of-the-art difficulty adaptation methods for such scenarios are based on task performance, which conveys little information about user-specific factors such as workload. Thus, we present an exploratory study of automated affect recognition and task difficulty adaptation in a competitive scenario based on physiological linkage (covariation of participants’ physiological responses). Classification algorithms were developed in an open-loop study where 16 pairs played a competitive game while 5 physiological responses were measured: respiration, skin conductance, electrocardiogram, and 2 facial electromyograms. Physiological and performance data were used to classify four self-reported variables (enjoyment, valence, arousal, perceived difficulty) into two or three classes. The highest classification accuracies were obtained for perceived difficulty: 84.3% for two-class and 60.5% for three-class classification. As a proof of concept, the developed classifiers were used in a small closed-loop study to dynamically adapt game difficulty. While this closed-loop study found no clear advantages of physiology-based adaptation, it demonstrated the technical feasibility of such real-time adaptation. In the long term, physiology-based task adaptation could enhance competition and cooperation in many multi-user settings (e.g., education, manufacturing, exercise).
Research Interests:
In recent years, several multi-user virtual environments (VEs) have been developed to promote motivation and exercise intensity in motor rehabilitation. While competitive VEs have been extensively evaluated, collaborative and competitive... more
In recent years, several multi-user virtual environments (VEs) have been developed to promote motivation and exercise intensity in motor rehabilitation. While competitive VEs have been extensively evaluated, collaborative and competitive rehabilitation VEs have seen relatively little study. Therefore, this paper presents an evaluation of a VE for post-stroke arm rehabilitation that mimics everyday kitchen tasks and can be used either solo or collaboratively. Twenty subacute stroke survivors exercised with the VE for four sessions, with the first and third sessions involving solo exercise and the other two involving collaborative exercise. Exercise intensity was measured using inertial sensors while motivation was measured with questionnaires. Results showed high motivation and exercise intensity over all four sessions, and 11 of 20 participants preferred collaborative over solo exercise while only 4 preferred solo exercise.However, there were no differences in motivation, exercise duration, or exercise intensity between solo and collaborative sessions. Thus,we cannot currently claim that collaborative exercises are beneficial for upper limb rehabilitation. Future studies should evaluate other collaborative VE designs in different settings (e.g., at home) and with different participant pairs (e.g., patient-unimpaired) to find effective ways to utilize collaborative exercises in motor rehabilitation.
Research Interests:
Objective: Trunk exoskeletons are a new technology with great promise for human rehabilitation, assistance and augmentation. However, it is unclear how different exoskeleton features affect the wearer's body during different activities.... more
Objective: Trunk exoskeletons are a new technology with great promise for human rehabilitation, assistance and augmentation. However, it is unclear how different exoskeleton features affect the wearer's body during different activities. This study thus examined how varying a trunk exoskeleton's thoracic and abdominal compression affects trunk kinematics and muscle demand during several activities. Methods: We developed a trunk exoskeleton that allows thoracic and abdominal compression to be changed quickly and independently. To evaluate the effect of varying compression, 12 participants took part in a two-session study. In the first session, they performed three activities (walking, sit-to-stand, lifting a box). In the second session, they experienced unexpected perturbations while sitting. This was done both without the exoskeleton and in four exoskeleton configurations with different thoracic and abdominal compression levels. Trunk flexion angle, low back extension moment and the electromyogram of the erector spinae and rectus abdominis were measured in both sessions. Results: Different exoskeleton compression levels resulted in significantly different peak trunk flexion angles and peak electromyograms of the erector spinae. However, the effects of compression differed significantly between activities. Conclusion: Our results indicate that a trunk exoskeleton's thoracic and abdominal compression affect the wearer's kinematics and muscle demand; furthermore, a single compression configuration is not appropriate for all activities. Significance: The study suggests that future trunk exoskeletons should either be able to vary their compression levels to suit different activities or should have the compression designed for a specific activity in order to be beneficial to the wearer.
Research Interests:
Although several studies have used wearable sensors to analyze human lifting, this has generally only been done in a limited manner. In this proof-of-concept study, we investigate multiple aspects of offline lift characterization using... more
Although several studies have used wearable sensors to analyze human lifting, this has generally only been done in a limited manner. In this proof-of-concept study, we investigate multiple aspects of offline lift characterization using wearable inertial measurement sensors: detecting the start and end of the lift and classifying the vertical movement of the object, the posture used, the weight of the object, and the asymmetry involved. In addition, the lift duration, horizontal distance from the lifter to the object, the vertical displacement of the object, and the asymmetric angle are computed as lift parameters. Twenty-four healthy participants performed two repetitions of 30 different main lifts each while wearing a commercial inertial measurement system. The data from these trials were used to develop, train, and evaluate the lift characterization algorithms presented. The lift detection algorithm had a start time error of 0.10 s ± 0.21 s and an end time error of 0.36 s ± 0.27 s across all 1489 lift trials with no missed lifts. For posture, asymmetry, vertical movement, and weight, our classifiers achieved accuracies of 96.8%, 98.3%, 97.3%, and 64.2%, respectively, for automatically detected lifts. The vertical height and displacement estimates were, on average, within 25 cm of the reference values. The horizontal distances measured for some lifts were quite different than expected (up to 14.5 cm), but were very consistent. Estimated asymmetry angles were similarly precise. In the future, these proof-of-concept offline algorithms can be expanded and improved to work in real-time. This would enable their use in applications such as real-time health monitoring and feedback for assistive devices.
Research Interests:
In dyadic motor learning, pairs of people learn the same motion while their limbs are loosely coupled together using haptic devices. Such coupled learning has been shown to outperform solo learning (including robot-guided learning) for... more
In dyadic motor learning, pairs of people learn the same motion while their limbs are loosely coupled together using haptic devices. Such coupled learning has been shown to outperform solo learning (including robot-guided learning) for simple one-degree-of-freedom tasks. However, results from more complex tasks are limited and sometimes conflicting. We thus evaluated coupled learning in a two-degree-of-freedom tracking task where participants also had to compensate for a simple force field. Participant pairs were split into two groups: an experiment group that experienced a compliant haptic coupling between participants' hands and a control group that did not. The study protocol consisted of 70 repetitions of 18.9-second tracking trials: 10 initial solo trials with no coupling, 50 "learning" trials (where participants in the experiment group were coupled), and 10 final solo trials with no coupling. The experiment group (coupled) improved their solo tracking performance both in the presence (p = 0.008) and absence (p < 0.001) of the force field; however, the control group (no coupling) only improved their solo performance in the absence of the force field (p < 0.001) but not in the presence of the field (p = 0.81). This suggests that dyadic motor learning can outperform solo learning for two-dimensional tracking motions in the presence of a simple force field, though the mechanism by which learning is improved is not yet clear. Clinical Relevance-As motor learning is critical for applications such as motor rehabilitation, dyadic training could be used to achieve a better overall outcome and a faster learning speed in these applications compared to solo training.
Research Interests:
Objective: Competitive exercise games are popular in areas like rehabilitation and weight loss due to their positive effects on motivation. However, it is unclear whether a human opponent is necessary, as the same benefits may be... more
Objective: Competitive exercise games are popular in areas like rehabilitation and weight loss due to their positive effects on motivation. However, it is unclear whether a human opponent is necessary, as the same benefits may be achievable with a ''human-like'' computer-controlled opponent or a human who talks to the player without playing the game. Our objective was to compare four opponent types in a competitive exercise game: a simple computer opponent, ''human-like'' computer opponent, human opponent, and a simple computer opponent accompanied by a player-selected human who chats with the player. Materials and Methods: Sixteen participants (3 women, 24.4-7.7 years old) played a competitive arm exercise game in the above four conditions. Exercise intensity was measured with inertial sensors, and four motivation scales were measured with the Intrinsic Motivation Inventory. After playing, participants answered several questions regarding their preferences. Results: The human opponent was the favorite for 14 of 16 participants and resulted in the highest interest/ enjoyment and exercise intensity. All participants preferred the human opponent over the computer opponent accompanied by a human companion. Finally, 12 of 16 participants preferred the ''human-like'' computer opponent over the simple one. Conclusion: Our results have two implications for competitive exercise games. First, they indicate that developing computer-controlled opponents with more human-like behavior is worthwhile, but that the best results are achieved with human opponents. Second, social interaction without in-game interaction does not provide an enjoyable, intense experience. However, our results should be verified with different target populations for exercise games.
Research Interests:
Background: Competitive and cooperative serious games have become increasingly popular in areas such as rehabilitation and education and have several potential advantages over single-player games. However, they are not suitable for... more
Background: Competitive and cooperative serious games have become increasingly popular in areas such as rehabilitation and education and have several potential advantages over single-player games. However, they are not suitable for everyone, and the user experience in competitive and cooperative serious games depends on many factors. One important factor is the verbal interaction between players, but the effect of this factor has not been extensively studied because of the lack of a validated measurement tool.
Research Interests: Interpersonal Communication, Serious Games, Conversation Analysis, Collaboration, Conversation Analysis (Research Methodology), and 9 moreMultiplayer Online Games, Computer Supported Collaborative Learning (CSCL), Serious Gaming, Competition, Interpersonal Psychology, Questionnaire design, Interpersonal Relationships, Multiplayer, and Questionnaires
Human psychological (cognitive and affective) dimensions can be assessed using several methods, such as physiological or performance measurements. To date, however, few studies have compared different data modalities with regard to their... more
Human psychological (cognitive and affective) dimensions can be assessed using several methods, such as physiological or performance measurements. To date, however, few studies have compared different data modalities with regard to their ability to enable accurate classification of different psychological dimensions. This study thus compares classification accuracies for four psychological dimensions and two subjective preferences about computer game difficulty using three data modalities: physiology, performance, and personality characteristics. Thirty participants played a computer game at nine difficulty configurations that were implemented via two difficulty parameters. In each configuration, seven physiological measurements and two performance variables were recorded. A short questionnaire was filled out to assess the perceived difficulty, enjoyment, valence, arousal, and the way the participant would like to modify the two difficulty parameters. Furthermore, participants' personality characteristics were assessed using four questionnaires. All combinations of the three data modalities (physiology, performance, and personality) were used to classify six dimensions of the short questionnaire into either two, three or many classes using four classifier types: linear discriminant analysis, support vector machine (SVM), ensemble decision tree, and multiple linear regression. The classification accuracy varied widely between the different psychological dimensions; the highest accuracies for two-class and three-class classification were 97.6 and 84.1%, respectively. Normalized physiological measurements were the most informative data modality, though current game difficulty, personality and performance also contributed to classification accuracy; the best selected features are presented and discussed in the text. The SVM and multiple linear regression were the most accurate classifiers, with regression being more effective for normalized physiological data. In the future, we will further examine the effect of different classification approaches on user experience by detecting the user's psychological state and adapting game difficulty in real-time. This will allow us to obtain a complete picture of the performance of affect-aware systems in both an offline (classification accuracy) and real-time (effect on user experience) fashion.
Research Interests:
Interpersonal rehabilitation games, which allow patients to compete or cooperate with other patients or unimpaired loved ones, have demonstrated promising short-term results, but have not yet been tested in longer-term studies. This paper... more
Interpersonal rehabilitation games, which allow patients to compete or cooperate with other patients or unimpaired loved ones, have demonstrated promising short-term results, but have not yet been tested in longer-term studies. This paper thus presents a preliminary 9-session evaluation of interpersonal rehabilitation games for post-stroke arm exercise. Two pairs of stroke survivors were provided with a system that included one competitive and one cooperative rehabilitation game, and exercised with it for 9 sessions in addition to their conventional therapy. They were able to choose the game they wanted to play in each session, and had to exercise for at least 10 minutes per session. Both pairs completed the protocol without any issues, reporting high levels of motivation and consistent levels of exercise intensity (measured using inertial sensors) across the sessions. Furthermore, the maximum difficulty levels reached in the cooperative game increased over time, and improvements of 1-8 points were observed on the Box and Block test. These results indicate that 2 different interpersonal games are sufficient to promote high levels of motivation and exercise intensity for 9 sessions performed over a 3-week period. As the next step, our system will be expanded with additional competitive, cooperative and single-player games, then tested in full clinical trials in both clinical and home environments.
Research Interests:
Competitive rehabilitation games can enhance motivation and exercise intensity compared to solo exercise; however, since such games may be played by two people with different abilities, game difficulty must be dynamically adapted to suit... more
Competitive rehabilitation games can enhance motivation and exercise intensity compared to solo exercise; however, since such games may be played by two people with different abilities, game difficulty must be dynamically adapted to suit both players. State-of-the-art adaptation algorithms are based on players' performance (e.g., score), which may not be representative of the patient's physical and psychological state. Instead, we propose a method that estimates players' states in a competitive game based on the covariation of players' physiological responses. The method was evaluated in 10 unimpaired pairs, who played a competitive game in 6 conditions while 5 physiological responses were measured: respiration, skin conductance, heart rate, and 2 facial electromyograms. Two physiological linkage methods were used to assess the similarity of the players' physiological measurements: coherence of raw measurements and correlation of heart and respiration rates. These linkage features were compared to traditional individual physiological features in classification of players' affects (enjoyment, valence, arousal, perceived difficulty) into 'low' and 'high' classes. Classifiers based on physiological linkage resulted in higher accuracies than those based on individual physiological features, and combining both feature types yielded the highest classification accuracies (75% to 91%). These classifiers will next be used to dynamically adapt game difficulty during rehabilitation.
Research Interests:
Trunk exoskeletons are an emerging technology that could reduce spinal loading, guide trunk motion, and augment lifting ability. However, while they have achieved promising results in brief laboratory studies, they have not yet been... more
Trunk exoskeletons are an emerging technology that could reduce spinal loading, guide trunk motion, and augment lifting ability. However, while they have achieved promising results in brief laboratory studies, they have not yet been tested in longer-term real-world studies-partially due to reliance on stationary sensors such as cameras. To enable future real-world evaluations of trunk exoskeletons, this paper describes two preliminary studies on using inertial measurement units (IMUs) to collect kinematic data from an exoskeleton wearer. In the first study, a participant performed three activities (walking, sit-to-stand, box lifting) while trunk flexion angle was measured with both IMUs and reference cameras. The mean absolute difference in flexion angle between the two methods was 1.4° during walking, 3.6° during sit-to-stand and 5.2° during box lifting, showing that IMUs can measure trunk flexion with a reasonable accuracy. In the second study, six participants performed five activities (standing, sitting straight, slouching, 'good' lifting, 'bad' lifting), and a naïve Bayes classifier was used to automatically classify the activity from IMU data. The classification accuracy was 92.2%, indicating the feasibility of automated activity classification using IMUs. The IMUs will next be used to obtain longer-term recordings of different activities performed both with and without a trunk exoskeleton to determine how the exoskeleton affects a person's posture and behavior.
Research Interests:
Drivers' hazardous physical and mental states (e.g., distraction, fatigue, stress, and high workload) have a major effect on driving performance and strongly contribute to 25-50% of all traffic accidents. They are caused by numerous... more
Drivers' hazardous physical and mental states (e.g., distraction, fatigue, stress, and high workload) have a major effect on driving performance and strongly contribute to 25-50% of all traffic accidents. They are caused by numerous factors, such as cell phone use or lack of sleep. However, while significant research has been done on detecting hazardous states, most studies have not tried to identify the causes of the hazardous states. Such information would be very useful, as it would allow intelligent vehicles to better respond to a detected hazardous state. Thus, this study examined whether the cause of a driver's hazardous state can be automatically identified using a combination of driver characteristics, vehicle kinematics, and physiological measures. Twenty-one healthy participants took part in four 45-min sessions of simulated driving, of which they were mildly sleep-deprived for two sessions. Within each session, there were eight different scenarios with different weather (sunny or snowy), traffic density and cell phone usage (with or without cell phone). During each scenario, four physiological (respiration, electrocardiogram, skin conductance, and body temperature) and eight vehicle kinematics measures were monitored. Additionally, three self-reported driver characteristics were obtained: personality, stress level, and mood. Three feature sets were formed based on driver characteristics, vehicle kinematics, and physiological signals. All possible combinations of the three feature sets were used to classify sleep deprivation (drowsy vs. alert), traffic density (low vs. high), cell phone use, and weather conditions (foggy/snowy vs. sunny) with highest accuracies of 98.8%, 91.4%, 82.3%, and 71.5%, respectively. Vehicle kinematics were most useful for classification of weather and traffic density while physiology and driver characteristics were useful for classification of sleep deprivation and cell phone use. Furthermore, a second classification scheme was tested that also incorporates information about whether or not other causes of hazardous states are present, though this did not result in higher classification accuracy. In the future, these classifiers could be used to identify both the presence and cause of a driver's hazardous state, which could serve as the basis for more intelligent intervention systems.
Research Interests:
Low back pain is a leading cause of disability, and there is a tremendous need for nonsurgical, nonpharmaceutical interventions to manage it. Versatile spinal exoskeletons have been proposed as a method of supporting or augmenting the... more
Low back pain is a leading cause of disability, and there is a tremendous need for nonsurgical, nonpharmaceutical interventions to manage it. Versatile spinal exoskeletons have been proposed as a method of supporting or augmenting the wearer, but experimental data from human subjects are limited, and the effects of such exoskeletons remain poorly understood. We thus present a prototype of a reconfigurable spinal exoskeleton that features easily adjustable resistance and compression at multiple spinal levels, allowing us to study the effect of different exoskeleton configurations on the body. In a pilot evaluation with a single subject, both thoracic and abdominal compression were found to affect trunk angle, low back moment and the electromyogram of the erector spinae, though different exoskeleton configurations had different effects during different tasks. This supports the premise that intelligent mechanical adjustments of a spinal exoskeleton are necessary for optimal support or augmentation of the wearer, though the results need to be examined in a larger, varied sample of subjects.
Research Interests:
Research Interests:
Motor rehabilitation technologies commonly include virtual environments that motivate patients to exercise more often or more intensely. In this paper, we present a novel virtual rehabilitation environment in which two people work... more
Motor rehabilitation technologies commonly include virtual environments that motivate patients to exercise more often or more intensely. In this paper, we present a novel virtual rehabilitation environment in which two people work together to prepare meals. The players' roles can be fixed or undefined, and optional challenges can be added in the form of flies that must be swatted away. A preliminary evaluation with 12 pairs of unimpaired participants showed that participants prefer cooperating over exercising alone and feel less pressured when cooperating. Furthermore, participants enjoyed the addition of flies and preferred not to have defined roles. Finally, no significant decrease in exercise intensity was observed as a result of cooperation. These results indicate that cooperation could improve motor rehabilitation by increasing motivation, though the virtual environment needs to be evaluated with participants with motor impairment.
Research Interests:
This paper presents a new approach to benchmarking brain-computer interfaces (BCIs) outside the lab. A computer game was created that mimics a real-world application of assistive BCIs, with the main outcome metric being the time needed to... more
This paper presents a new approach to benchmarking brain-computer interfaces (BCIs) outside the lab. A computer game was created that mimics a real-world application of assistive BCIs, with the main outcome metric being the time needed to complete the game. This approach was used at the Cybathlon 2016, a competition for people with disabilities who use assistive technology to achieve tasks. The paper summarizes the technical challenges of BCIs, describes the design of the benchmarking game, then describes the rules for acceptable hardware, software and inclusion of human pilots in the BCI competition at the Cybathlon. The 11 participating teams, their approaches, and their results at the Cybathlon are presented. Though the benchmarking procedure has some limitations (for instance, we were unable to identify any factors that clearly contribute to BCI performance), it can be successfully used to analyze BCI performance in realistic, less structured conditions. In the future, the parameters of the benchmarking game could be modified to better mimic different applications (e.g., the need to use some commands more frequently than others). Furthermore, the Cybathlon has the potential to showcase such devices to the general public.
Research Interests:
Background: People with neurological injuries such as stroke should exercise frequently and intensely to regain their motor abilities, but are generally hindered by lack of motivation. One way to increase motivation in rehabilitation is... more
Background: People with neurological injuries such as stroke should exercise frequently and intensely to regain their motor abilities, but are generally hindered by lack of motivation. One way to increase motivation in rehabilitation is through competitive exercises, but such exercises have only been tested in single brief sessions and usually did not adapt difficulty to the patient's abilities. Methods: We designed a competitive arm rehabilitation game for two players that dynamically adapts its difficulty to both players' abilities. This game was evaluated by two participant groups: 15 participants with chronic arm impairment who exercised at home with an unimpaired friend or relative, and 20 participants in the acute or subacute phase of stroke who exercised in pairs (10 pairs) at a rehabilitation clinic. All participants first played the game against their human opponent for 3 sessions, then played alone (against a computer opponent) in the final, fourth session. In all sessions, participants' subjective experiences were assessed with the Intrinsic Motivation Inventory questionnaire while exercise intensity was measured using inertial sensors built into the rehabilitation device. After the fourth session, a final brief questionnaire was used to compare competition and exercising alone. Results: Participants who played against an unimpaired friend or relative at home tended to prefer competition (only 1 preferred exercising alone), and exhibited higher enjoyment and exercise intensity when competing (first three sessions) than when exercising alone (last session). Participants who played against each other in the clinic, however, did not exhibit significant differences between competition and exercising alone. For both groups, there was no difference in enjoyment or exercise intensity between the first three sessions, indicating no negative effects of habituation or novelty. Conclusions: Competitive exercises have high potential for unsupervised home rehabilitation, as they improve enjoyment and exercise intensity compared to exercising alone. Such exercises could thus improve rehabilitation outcome, but this needs to be tested in long-term clinical trials. It is not clear why participants who competed against each other at the clinic did not exhibit any advantages of competition, and further studies are needed to determine how different factors (environment, nature of opponent etc.) influence patients' experiences with competitive exercises.
Research Interests:
Rehabilitation robots are often combined with serious games that motivate patients and keep them exercising at high intensities. A promising type of game are competitive rehabilitation games, but few difficulty adaptation algorithms have... more
Rehabilitation robots are often combined with serious games that motivate patients and keep them exercising at high intensities. A promising type of game are competitive rehabilitation games, but few difficulty adaptation algorithms have been presented for them. This paper thus presents the adaptation of difficulty in a competitive arm rehabilitation game based on two physiological signals: respiration and electromyography of the posterior deltoid. It consists of three smaller studies: an open-loop respiration study, a closed-loop respiration study (where a controller attempts to maintain respiration rate at preset levels), and a closed-loop electromyogram study (where a controller attempts to keep the electromyogram at preset levels). The studies control two difficulty parameters based on the physiological responses of one of the two exercising participants, though the ultimate goal is to control the physiological responses of both participants. Furthermore, all three studies are done with unimpaired participants. The closed-loop controllers achieved high correlation coefficients between desired and measured levels of respiration rate (r = 0.83) and electromyogram (r = 0.89), demonstrating that it is possible to control the physiological responses of unimpaired participants in a competitive arm rehabilitation game, thus controlling their level of workload and exercise intensity. In the future, the proposed method will be tested with patients undergoing rehabilitation.
Research Interests:
This paper presents two different strategies for difficulty adaptation in a competitive arm rehabilitation game: a manual adaptation strategy and an automatic performance-based adaptation strategy. The two strategies were implemented in a... more
This paper presents two different strategies for difficulty adaptation in a competitive arm rehabilitation game: a manual adaptation strategy and an automatic performance-based adaptation strategy. The two strategies were implemented in a competitive game controlled with an inertial-sensor-based home rehabilitation device. They were first evaluated with 32 pairs of unimpaired participants, who played the game with manual adaptation, automated adaptation, or no adaptation. Each variant was played for 9 minutes. Then, the manual and automatic adaptation were also tested by 5 pairs consisting of a person with arm impairment (due to neurological injury) and their unimpaired friend or relative. Throughout the game, motivation was measured with questionnaires while exercise intensity was tracked using the inertial sensors. Results showed that both manual and automatic difficulty adaptation lead to higher motivation and exercise intensity than no adaptation. Unimpaired participants showed no clear preference between manual and automatic adaptation while 4 of 5 impaired participants preferred automatic adaptation. For future use, we propose a combination of manual and automatic adaptation that should be evaluated with more impaired participants in longer multisession experiments.
Research Interests:
Background: People with chronic arm impairment should exercise intensely to regain their abilities, but frequently lack motivation, leading to poor rehabilitation outcome. One promising way to increase motivation is through interpersonal... more
Background: People with chronic arm impairment should exercise intensely to regain their abilities, but frequently lack motivation, leading to poor rehabilitation outcome. One promising way to increase motivation is through interpersonal rehabilitation games, which allow patients to compete or cooperate together with other people. However, such games have mainly been evaluated with unimpaired subjects, and little is known about how they affect motivation and exercise intensity in people with chronic arm impairment.
Methods: We designed four different arm rehabilitation games that are played by a person with arm impairment and their unimpaired friend, relative or occupational therapist. One is a competitive game (both people compete against each other), two are cooperative games (both people work together against the computer) and one is a single-player game (played only by the impaired person against the computer). The games were played by 29 participants with chronic arm impairment, of which 19 were accompanied by their friend or relative and 10 were accompanied by their occupational therapist. Each participant played all four games within a single session. Participants' subjective experience was quantified using the Intrinsic Motivation Inventory questionnaire after each game, as well as a final questionnaire about game preferences. Their exercise intensity was quantified using wearable inertial sensors that measured hand velocity in each game.
Methods: We designed four different arm rehabilitation games that are played by a person with arm impairment and their unimpaired friend, relative or occupational therapist. One is a competitive game (both people compete against each other), two are cooperative games (both people work together against the computer) and one is a single-player game (played only by the impaired person against the computer). The games were played by 29 participants with chronic arm impairment, of which 19 were accompanied by their friend or relative and 10 were accompanied by their occupational therapist. Each participant played all four games within a single session. Participants' subjective experience was quantified using the Intrinsic Motivation Inventory questionnaire after each game, as well as a final questionnaire about game preferences. Their exercise intensity was quantified using wearable inertial sensors that measured hand velocity in each game.
Research Interests:
Classification algorithms are used in affective computing to classify the state of the user and adapt the computer's behaviour, but it is unclear how classification accuracy influences the overall user experience. We present a study in... more
Classification algorithms are used in affective computing to classify the state of the user and adapt the computer's behaviour, but it is unclear how classification accuracy influences the overall user experience. We present a study in which classification accuracy is artificially pre-defined and used to adapt to the difficulty of a video game. Eighty subjects played the game and were told that difficulty would be adapted according to the measured brain activity. They played the game twice, with different classification accuracies, and then reported different aspects of their overall game experience using questionnaires. Classification accuracy was correlated with both in-game fun (r = 0.46) and satisfaction with the difficulty adaptation (r = 0.56). Most subjects could perceive a difference between two classification accuracies that differed by 16.7%. We tentatively posit that, for affective video games, an acceptable classification accuracy is 70–80%. Furthermore, studies that attempt to improve affect classification accuracy should aim for a practically meaningful improvement of 10%. RESEARCH HIGHLIGHTS • Affect classification accuracy is significantly correlated with the fun experienced in an affective video game. • Classification accuracies below 70% are unacceptable to end users while increasing accuracy above 90% has only small benefits. • Users are unlikely to perceive changes in classification accuracy that are smaller than 10%. Therefore, developers who want to improve affect recognition in their game should aim for a practically meaningful improvement of 10%.
Research Interests:
Physiological games use classification algorithms to extract information about the player from physiological measurements and adapt game difficulty accordingly. However, little is known about how the classification accuracy affects the... more
Physiological games use classification algorithms to extract information about the player from physiological measurements and adapt game difficulty accordingly. However, little is known about how the classification accuracy affects the overall user experience and how to measure this effect. Following up on a previous study, we artificially predefined classification accuracy in a game of Snake where difficulty increases or decreases after each round. The game was played in a laboratory setting by 110 participants at different classification accuracies. The participants reported their satisfaction with the difficulty adaptation algorithm as well as their in-game fun, with 85 participants using electronic questionnaires and 25 using paper questionnaires. We observed that the classification accuracy must be at least 80% for the physiological game to be accepted by users and that there are notable differences between different methods of measuring the effect of classification accuracy. The results also show that laboratory settings are more effective than online settings, and paper questionnaires exhibit higher correlations between classification accuracy and user experience than electronic questionnaires. Implications for the design and evaluation of physiological games are presented.
Research Interests:
Modern wearable robots are not yet intelligent enough to fully satisfy the demands of end users, as they lack the sensor fusion algorithms needed to provide optimal assistance and react quickly to perturbations or changes in user... more
Modern wearable robots are not yet intelligent enough to fully satisfy the demands of end users, as they lack the sensor fusion algorithms needed to provide optimal assistance and react quickly to perturbations or changes in user intentions. Sensor fusion applications such as intention detection have been emphasized as a major challenge for both robotic orthoses and prostheses. In order to better examine the strengths and shortcomings of the field, this paper presents a review of existing sensor fusion methods for wearable robots, both stationary ones such as rehabilitation exoskeletons and
portable ones such as active prostheses and full-body exoskeletons. Fusion methods are first presented as applied to individual sensing modalities (primarily electromyography, electroencephalography and mechanical sensors), and then four approaches to combining multiple modalities are presented. The strengths and weaknesses of the different methods are compared, and recommendations are made for
future sensor fusion research.
portable ones such as active prostheses and full-body exoskeletons. Fusion methods are first presented as applied to individual sensing modalities (primarily electromyography, electroencephalography and mechanical sensors), and then four approaches to combining multiple modalities are presented. The strengths and weaknesses of the different methods are compared, and recommendations are made for
future sensor fusion research.
Research Interests:
This paper presents a teleoperation system for two six-degree-of-freedom ARMin arm rehabilitation skeletons. Two control strategies are presented: unilateral teleoperation, which is realized with proportional-derivative controllers, and... more
This paper presents a teleoperation system for two six-degree-of-freedom ARMin arm rehabilitation skeletons. Two control strategies are presented: unilateral teleoperation, which is realized with proportional-derivative controllers, and bilateral teleoperation, which is realized with compliance control and torque feedback. An empirical procedure for identifying the best control gains of the unilateral teleoperation system is presented, and the stability of the bilateral teleoperation system is demonstrated theoretically. Evaluations with healthy subjects showed that trajectories can be learned through the teleoperation system and that the torque feedback allows one human user to determine whether the other human is participating in the movement, resisting it, or remaining passive. The paper concludes with a discussion of potential applications of the system, including telerehabilitation and group rehabilitation.
Research Interests:
Physiological computing represents a mode of human–computer interaction where the computer monitors, analyzes and responds to the user’s psychophysiological activity in real-time. Within the field, autonomic nervous system responses have... more
Physiological computing represents a mode of human–computer interaction where the computer monitors, analyzes and responds to the user’s psychophysiological activity in real-time. Within the field, autonomic nervous system responses have been studied extensively since they can be measured quickly and unobtrusively. However, despite a vast body of literature available on the subject, there is still no universally accepted set of rules that would translate physiological data to psychological states. This paper surveys the work performed on data fusion and system adaptation using autonomic nervous system responses in psychophysiology and physiological computing during the last ten years. First, five prerequisites for data fusion are examined: psychological model selection, training set preparation, feature extraction, normalization and dimension reduction. Then, different methods for either classification or estimation of psychological states from the extracted features are presented and compared. Finally, implementations of system adaptation are reviewed: changing the system that the user is interacting with in response to cognitive or affective information inferred from autonomic nervous system responses. The paper is aimed primarily at psychologists and computer scientists who have already recorded autonomic nervous system responses and now need to create algorithms to determine the subject’s psychological state.
Research Interests:
Several strategies have been proposed to improve patient motivation and exercise intensity during robot-aided stroke rehabilitation. One relatively unexplored possibility is two-player gameplay, allowing subjects to compete or cooperate... more
Several strategies have been proposed to improve patient motivation and exercise intensity during robot-aided stroke rehabilitation. One relatively unexplored possibility is two-player gameplay, allowing subjects to compete or cooperate with each other to achieve a common goal. In order to explore the potential of such games, we designed a two-player game played using two ARMin arm rehabilitation robots.
Research Interests:
In an affective feedback loop, the computer maps various measurements to affective variables such as enjoyment, then adapts its behavior based on the recognized affects. The affect recognition is never perfect, and its accuracy... more
In an affective feedback loop, the computer maps various measurements to affective variables such as enjoyment, then adapts its behavior based on the recognized affects. The affect recognition is never perfect, and its accuracy (percentage of times the correct affective state is recognized) depends on many factors. However, it is unclear how this accuracy relates to the overall user experience. As recognition accuracy is difficult to control in a real affective feedback loop, we describe a method of simulating recognition accuracy in a game where difficulty is increased or decreased after each round. The game was played by 261 participants at different simulated recognition accuracies. Participants reported their satisfaction with the recognition algorithm as well as their overall game experience. We observed that in such a game, the affective feedback loop must adapt game difficulty with an accuracy of at least 80% to be accepted by users. Furthermore, users who do not enjoy the game are likely to stop playing it rather than continue playing and report low enjoyment. However, the acceptable recognition accuracy may not generalize to other contexts, and studies of affect recognition accuracies in other applications are needed.
Research Interests:
This paper uses physiological measurements to estimate human workload and effort in physical human–robot interaction. Ten subjects performed 19 consecutive task periods using the ARMin robot while difficulty was varied along two scales.... more
This paper uses physiological measurements to estimate human workload and effort in physical human–robot interaction. Ten subjects performed 19 consecutive task periods using the ARMin robot while difficulty was varied along two scales. Three physiological modalities were measured: electroencephalography, autonomic nervous system (ANS) responses (electrocardiography, skin conductance, respiration, skin temperature) and eye tracking. After each task period, reference workload and effort values were collected using the NASA Task Load Index. Machine learning was used to estimate workload and effort from physiological data. All three physiological modalities performed significantly better than random, particularly using nonlinear estimation algorithms. The most important ANS responses were respiration and skin conductance, while the most important electroencephalographic information was obtained from frontal and central sites. However, all three physiological modalities were outperformed by task performance and movement data. This suggests that future studies should try to demonstrate advantages of physiological measurements over other information sources.
Research Interests:
Research Interests:
Rapid recognition of voluntary motions is crucial in human-computer interaction, but few studies compare the predictive abilities of different sensing technologies. This paper thus compares performances of different... more
Rapid recognition of voluntary motions is crucial in human-computer interaction, but few studies compare the predictive abilities of different sensing technologies. This paper thus compares performances of different technologies when predicting targets of human reaching motions: electroencephalography, electrooculography, camera-based eye tracking, electromyography, hand position and the user’s preferences. Supervised machine learning is used to make predictions at different points in time (before and during limb motion) with each individual sensing modality. Different modalities are then combined using an algorithm that takes into account the different times at which modalities provide useful information. Results show that electroencephalography can make predictions before limb motion onset, but requires subject-specific training and exhibits decreased performance as the number of possible targets increases. Electromyography and hand position give high accuracy, but only once the motion has begun. Eye tracking is robust and exhibits high accuracy at the very onset of limb motion. Several advantages of combining different modalities are also shown, including advantages of combining measurements with contextual data. Finally, some recommendations are given for sensing modalities with regard to different criteria and applications. The information could aid human-computer interaction designers in selecting and evaluating appropriate equipment for their applications.
Research Interests:
Research Interests:
Abstract This paper presents a novel multimodal virtual rehabilitation environment. Its design and implementation are based on principles related to intrinsic motivation and game design. The system consists of visual, acoustic, and haptic... more
Abstract This paper presents a novel multimodal virtual rehabilitation environment. Its design and implementation are based on principles related to intrinsic motivation and game design. The system consists of visual, acoustic, and haptic modalities. Elements contributing to intrinsic motivation are carefully joined in the three modalities to increase patients' motivation during the long process of rehabilitation.
Research Interests:
Research Interests:
Research Interests:
Research Interests:
Abstract This paper examines the psychophysiological effects of mental workload in single-task and dual-task human-computer interaction. A mental arithmetic task and a manual error correction task were performed both separately and... more
Abstract This paper examines the psychophysiological effects of mental workload in single-task and dual-task human-computer interaction. A mental arithmetic task and a manual error correction task were performed both separately and concurrently on a computer using verbal and haptic input devices. Heart rate, skin conductance, respiration and peripheral skin temperature were recorded in addition to objective performance measures and self-report questionnaires.
This paper presents the MIMICS MMS rehabilitation system with a virtual rehabilitation task that includes several modes of haptic assistance. We observed the influence of these different modes of assistance on task performance and work... more
This paper presents the MIMICS MMS rehabilitation system with a virtual rehabilitation task that includes several modes of haptic assistance. We observed the influence of these different modes of assistance on task performance and work performed toward the target during the pick-and-place movement. Twenty-three hemiparetic subjects and a control group of twenty-three subjects participated in the study. The haptic assistance resulted in improved task performance and lower work performed during pick-and-place movement.
Abstract We present an automated segmentation method based on the analysis of plantar pressure signals recorded from two synchronized wireless foot insoles. Given the strict limits on computational power and power consumption typical of... more
Abstract We present an automated segmentation method based on the analysis of plantar pressure signals recorded from two synchronized wireless foot insoles. Given the strict limits on computational power and power consumption typical of wearable electronic components, our aim is to investigate the capability of a Hidden Markov Model machine-learning method, to detect gait phases with different levels of complexity in the processing of the wearable pressure sensors signals.