1. Study Description
This study investigates the development and application of a haptic interface system utilizing Transcranial Magnetic Stimulation (TMS) to generate virtual touch sensations within a Virtual Reality (VR) environment. The primary research focus revolves around identifying risks, technical challenges, and ethical concerns linked to employing TMS for haptic feedback. Detailed information about the use of TMS for haptics may be found in the Appendix.
Introduction
This case study explores the development, implementation, and application of a haptic interface system utilizing Transcranial Magnetic Stimulation (TMS) to generate virtual touch sensations. The research project is led by the University of Chicago in the Computer Science department, aimed at exploring non-invasive brain stimulation to enhance virtual reality (VR) experiences through advanced haptic feedback.
Research Objective
The primary objective of this research is to overcome the limitations of traditional haptic devices, which are often cumbersome and require multiple actuators attached to various parts of the body. The central hypothesis posits that a single, centralized TMS-based system can stimulate the brain's sensorimotor cortex to produce full-body haptic sensations. This approach aims to provide a more immersive VR experience by intercepting the nervous system at the source, the brain, to deliver haptic feedback without the need for external physical devices, like vibrotactile stimulators.
What is Transcranial Magnetic Stimulation (TMS)?
TMS is a non-invasive form of brain stimulation that uses magnetic fields to stimulate nerve cells in the brain. Developed in the mid-1980s, TMS has been widely used in both research and clinical settings. TMS works by creating a magnetic pulse that induces an electrical current in the targeted area of the brain. This current can either stimulate or inhibit activity in neurons, depending on the frequency and intensity of the magnetic pulses.
Clinical Use. TMS has been utilized in several medical fields, especially in psychiatry and neurology. In clinical therapy, repetitive TMS (rTMS) is commonly employed to treat depression, addiction, anxiety, and other mental health conditions. By targeting specific areas of the brain, TMS can modulate brain activity, providing relief for patients whose conditions are resistant to medication.
TMS therapy typically involves sessions spread out over several weeks. A standard treatment might involve daily sessions for 20 days, or intensive sessions lasting 6 minutes every hour for an entire week, adding up to as many as 18,000 pulses during the course of treatment. The frequency and intensity of these pulses are calibrated according to the patient’s specific needs and condition.
Research Use. Beyond clinical applications, TMS is widely used in research to study brain functions, cognition, and neural pathways. By selectively stimulating or inhibiting specific areas of the brain, researchers can explore brain-behavior relationships. For example, TMS has been used to temporarily 'turn off' certain brain regions to study their role in motor skills, memory, and sensory perception.
Ethical Challenges
The open-ended query for this case study seeks to explore the potential risks associated with TMS, including safety protocols, user experiences, and the technological limitations of applying TMS in immersive VR environments.
- What ethical challenges do you think arise in haptics and brain stimulation research? How does the added layer VR impact that assessment? Do you think these technologies are suitable for non-medical research purposes?
- Do you think standard informed consent protocols comprehensive enough to cover the novel risks introduced by combining TMS with immersive technologies?
Collect your group's thoughts on the Open Inquiry Notes
2. Technical Description
The technical foundation of this study lies in TMS technology, which stimulates the sensorimotor cortex of the brain to generate full-body haptic feedback, and robotic control over TMS positioning, and synchronous control system from the VR application to TMS activation. This process involves several inherent risks that need to be carefully managed.
Figure 1. (a) Source-haptics apparatus for interactive VR experience. Implemented by mechanically moving a coil across the user’s scalp. Types of haptic interactions: (b) recoil of throwing a projectile, (c) impact on the leg, (d) force of stomping on a box, (e), impact of a projectile on one’s hand, or (f) an explosion close to the jaw. Source: https://doi.org/10.1145/3613904.3642483
Experimental Design
The experimental design involved a two-part study: calibration and user experience. Both studies were governed by a code of conduct prioritizing participant safety and anonymity
- Calibration Study: The first study focused on characterizing the range of haptic sensations that can be induced by stimulating different areas of the sensorimotor cortex. A grid search technique was employed to determine the motor and sensory thresholds for various body parts. This calibration was critical for setting the parameters required for effective haptic feedback.
- User Studies: The second study involved an empirical evaluation of user experiences with the TMS-based system integrated into a VR environment. Participants engaged with a custom-built VR game, and the TMS system was used to simulate touch sensations in real-time, allowing researchers to assess the system’s impact on the overall VR experience.
Safety Protocols
The study team developed the following safety protocols.
Stimulation Parameters: The TMS system was programmed to follow safety guidelines, particularly concerning the frequency and intensity of stimulation. Pulses were separated by at least two to three seconds to prevent overstimulation, with software safety measures ensuring that no two pulses occurred within a five-second window.
Participant Screening: Participants were carefully screened to exclude those with metal implants or other contraindications to TMS. During the study, all participants were required to remove any metallic objects, such as earrings, to avoid interference with the magnetic fields.
Device Interference: The study tested for potential interference between the TMS system and other electronic devices, such as the VR headset. It was confirmed that the distance and shielding of devices, such as the Meta Quest 2, minimized any potential electromagnetic interference, ensuring both the safety of participants and the integrity of the data collected.
System Implementation
The TMS-based haptic interface system was developed as an engineering prototype capable of delivering targeted stimulation to the brain’s sensorimotor cortex. The system included a robotic platform to move the TMS coil, controlled through an interface integrated with the VR environment. The system was calibrated to the user’s predefined sensorimotor map with coil’s movements of 2-3 mm precision. The haptic feedback was synchronized with the user’s actions within the VR space with a latency of 16 ms.
Game Development
To evaluate the effectiveness of the TMS-based haptic feedback system, a custom VR escape game was developed using Unity. This game required participants to interact with various virtual objects, triggering haptic sensations through the TMS system. The game’s design included elements that would naturally require touch interactions, such as manipulating objects and navigating through the virtual environment. The integration of the TMS API with Unity allowed for real-time control of the haptic feedback. An emergency stop function from the application would power down the TMS and robotic actuators.
Technical Analysis
- How does the study mitigate the risks of electromagnetic interference that could affect both user safety and data integrity?
- Are there mechanisms to monitor and address any unintended psychological impacts of the TMS?
Collect your group's thoughts on the Technical Analysis Notes
3. Risk Analysis
IRB Protocols
The study protocol was reviewed by the university the Institutional Review Board (IRB), which categorized the study under Behavioral Science rather than Biomedical Research. This determined the scope of the review and the committee composition. The PI was a Computer Science professor. The IRB protocols included:
- Full Review: Given the novel use of TMS in a non-traditional application, the study underwent a full IRB review.
- Informed Consent: Participants were provided with standard informed consent forms. Audio and video recordings were collected. Interview transcripts were recorded. Adverse effects of TMS were listed.
- Participant Welfare: In the study, the welfare of participants was monitored by verbal questions by the experimenter. Participants were reminded they could withdraw at any time.
- Compensation: Participants were given a $30 Amazon gift card at the completion of the study.
- Data Management Plan: All collected data, including video recordings of participant interactions, were securely stored on a password-protected university server. The protocol mandated that all local copies of the data be deleted after successful upload to ensure data security.
Review Concerns
Approval of this haptics study was built upon the previous of TMS research at the university in a different department. To link this continuum of research, experimenters from the new lab received training from the established lab. Further, all study personnel were required to attain First Aid certification.
Calibration was used to determine the minimal amount of stimulation to produce the desired effect. Calibration of the stimulation parameters for brief sensory effects and not motor effects would reduce the likelihood of adverse events. TMS has the potential to cause nausea, temporary paralysis of movement, and with repeated use change in brain plasticity. Experimentors are not permitted to exceed the parameters or change the locations of stimulation after calibration.
Risk Assessment
- What are the key risk to participant safety in this study? How did the review process address these risks?
- Was the level of review and committee type appropriate for the nature of this research?
- Was the informed consent as described sufficient to educate the participants prior to enrolling in the study?
Collect your group's thoughts on the Risk Assessment Notes
4. Conclusion
This research contributes to the field of haptic technology by demonstrating the feasibility of using TMS to create full-body haptic sensations in a VR environment. The paradigm simulated body region specific and tactile sensation types by direct brain stimulation. For example, stepping on a virtual block stimulated force feedback on the foot. The holistic experience is the block pushing back on the sole of the foot. Source-haptics would change the scope of the embodiment of immersive experiences and brain-computer-interfaces. This study represented the seminal use of TMS in VR applications.
Mitigation Strategy
(Image: Table with column headers Risk Category, Description, Stakeholder Responsible, Mitigation Strategies with respective lists of Physical Safety, Data Use, Informed Consent, then TMS has non-minimal risk of discomfort (e.g. nausea), Current and Future, Clearly communicating safety risks, then PI/ Experimenters/Health Unit, PI/IRB, PI/IRB, then Training for Experimenters and restricted use, Report planned and potential analytics, Procedures to allow for participant withdrawal. Source: Scott, Chaudhary, Bagade, 2024.)
Mitigation
- What was the most important action by the researchers and IRB to protect participants from undue risk? Would you recommend additional steps or a different approach?
- Do you think brain stimulation and virtual reality research is appropriate to conduct in a Computer Science department lab setting?
Collect your group's thoughts on the Mitigation Strategy Notes
Acknowledgments
The research team would like to acknowledge the University of Chicago research group that shared their study materials and consulted for this project.
Appendix
Comparison of Past Research on Haptics and the Present Study
|
Research Methodology |
Description |
Challenges/Limitations |
Present Study (TMS-based Haptics) |
|---|---|---|---|
|
Full-body Haptic Feedback via Endpoint |
Attaching multiple actuators to the body (e.g., vibrotactile actuators, solenoids) to deliver sensations at specific body parts, like hands or feet. |
Requires many actuators, cumbersome setup, limits mobility, difficult to reconfigure actuator placement based on user needs. |
Uses a single TMS device to stimulate the brain’s sensorimotor cortex, providing full-body feedback without body-attached devices, simplifying setup and mobility. |
|
Peripheral Nerve Stimulation (Midpoint) |
Stimulates nerves away from the intended sensation location (e.g., wrist for hand sensations), producing referred tactile or force feedback sensations. |
Limited ability to provide diverse feedback, requires external electrodes to be attached to the body, limits flexibility for full-body sensations. |
TMS stimulates the brain directly, allowing for more flexibility in the variety and location of haptic feedback, including hands, arms, legs, and other areas without multiple body connections. |
|
Electrical Muscle Stimulation (EMS) |
Stimulates muscles to induce movement or force feedback. Electrodes placed on the skin cause muscles to contract, creating movement in limbs or other body parts. |
Limited to inducing movement rather than detailed tactile feedback, requires external electrodes and precise calibration to avoid discomfort or unintended movements. |
TMS provides both force and tactile feedback without requiring body-worn electrodes. Allows for more comprehensive and natural sensations with minimal setup time. |
|
TMS in Neuroscience |
Transcranial Magnetic Stimulation has been used in neuroscience research to study brain functions, such as motor control, vision, and language processing. |
Not traditionally used for interactive haptic systems. Limited in terms of providing real-time sensory feedback. |
This study extends the use of TMS to interactive haptics, utilizing its ability to stimulate the sensorimotor cortex for full-body haptic feedback, making it one of the first implementations in this area. |
Core Elements of TMS in Research and Therapy
|
Technology Background |
Clinical TMS |
Haptics TMS |
|---|---|---|
|
Transcranial Magnetic Stimulation (TMS) is a non-invasive method of stimulating the brain by using electromagnetic fields. This technology is frequently applied in neurological and psychological therapies to modify brain activity, with specific applications in treating conditions like depression and addiction. |
In clinical settings, repeated TMS sessions are typically 1 hour sessions, repeated daily, totaling thousands of pulses of stimulation. (e.g. 90,000 pulses over 4-6 weeks with a frequency of 10Hz and 120% motor threshold intensity) |
However, in this research study, participants <15 bursts, an amount negligible compared to therapeutic treatments. Hence, the stimulation applied during the study did not have any lasting or significant effects on brain function. |
|
Stimulation Frequency |
10-50Hz stimulation range with the use of burst delivery |
3 pulse burst with 5 second intervals |
|
Inter-stimulation Interval |
20 to 50 sessions |
Trains of stimulation separated by a minimum interval of 5 seconds |
Comparison Between TMS and Vagal Nerve Stimulation
|
TMS (Transcranial Magnetic Stimulation) |
tVNS (Transcutaneous Vagal Nerve Stimulation |
|---|---|
|
Non-invasive brain stimulation technique that uses electromagnetic pulses to stimulate neurons in the brain. It is applied externally via a coil placed on the scalp, targeting specific brain regions such as the sensorimotor cortex. |
Non-invasive form of vagal nerve stimulation that does not require surgical implantation. Instead, it stimulates the vagus nerve through the skin, usually at the ear or neck, using an external device. |
|
Used in research and clinical therapy to modulate brain function without requiring surgery. In experimental studies, it can provide haptic sensations by stimulating the brain’s sensorimotor cortex. |
Primarily used in research and therapeutic applications, often as a treatment for conditions such as epilepsy or depression. The stimulation is applied via electrodes attached to the skin, making it more accessible and less invasive. |
|
TMS does not require any invasive procedure, making it safer for short-term use in experiments. It is controlled externally and delivers targeted brain stimulation through a magnetic coil. |
tVNS allows for temporary and adjustable stimulation of the vagus nerve, which can be controlled externally. It does not require surgery, making it suitable for short-term experimental use. |
|
Used to elicit full-body haptic sensations in VR environments without the need for body-attached actuators. The stimulation focuses on brain regions that control sensations in hands, feet, and other body parts. |
tVNS does not typically provide haptic feedback or interact directly with sensory perception. Instead, it modulates mood or brain activity |
References
Tanaka, J. Serfaty, and P. Lopes, “Haptic Source-Effector: Full-Body Haptics via Non-Invasive Brain Stimulation,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, Honolulu HI USA: ACM, May 2024, pp. 1–15. doi: 10.1145/3613904.3642483.
