Extended reality (XR) is an umbrella term of Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). It refers to all real-and-virtual combined environments generated by computer. Users can immerse themselves in and communicate with the environments. XR is a disruptive technology may significantly change the horizon of patient care, medical research and education.
VR is a 100% virtual environment in which users can explore and interact with objects, which can be anatomic models reconstructed from CT, MRI or Ultrasound images. We first used VR technology (Valve Index) to observe patient’s complex fractured musculoskeletal structures since trauma surgeons reported the turnaround time of printing an anatomic model (around one day) is too long, and VR only takes 2-4 hours. Eventually we added more features and allowed surgeons to plan and simulate surgeries preoperatively.
AR superimposes virtual models to the real world, often done by using the camera on a smartphone or tablet. MR blends elements of both AR and VR, where physical and virtual objects co-exist and the user can interact with both of them in real-time. MR headset (Hololens 2) is easier to use than VR system since it recognizes hand gestures so the user can interact with the virtual world with hands, while VR can only use controllers. Initially we transferred VR based anatomic model observation and surgery simulation to a MR platform, then we expand it to other applications. In a recent research we use Hololens’ object tracking feature to develop an App that can project blood vessels and tumors onto a physical liver model so a surgeon can see through the tissue and avoid important structures when resecting tumors. We found a great potential of MR in medical training. We developed an APP for medical students to survey a vital patient and conduct measurements in a visiting room combined with virtual and real instrument, furniture and characters. We are also exploring the application of MR in telemedicine, telesurgery, and teleconference.
We also are developing telecommunication and telemedicine capacities for physicians and patients. Right now we are able to transfer and store XR models/Apps through high-speed internet and the cloud so surgeons can observe and manipulate patient models at their own office or anywhere with an XR headset. We are also exploring technological details of teleconference and teleconsultation.
