“Immersive Layers” – How Apple’s Vision Pro will revolutionize medicine
Rafael Fonseca, M.D.1
Ellen Meltzer, M.D.1
Jennie Kung, M.B.A.1
Aman Anand, Ph.D.1
1 Mayo Clinic in Arizona, Phoenix, AZ
Introduction
Immersive layering
On February 2nd 2024 Apple released[1] the Apple Vision Pro™ (AVP) product – i.e., spatial computing that allows a simultaneous visual experience of reality, with superimposed layers of information. These layers are derived from the internal computer of the AVP (and its connectivity), augmenting the availability of visual and auditory information available to the user. In this article, we will explore some of the hypothetical uses of such devices in medicine and why we think this will be the first step in a transformative process to increase the quality of medicine. Devices like the AVP will create efficiencies, improve the experiences of providers, learners, patients, and caregivers, and will ultimately transform the provision of care. Much like the electronic medical record transformed the documentative aspect of medicine (and other aspects like electronic prescriptions), we argue that devices such as the AVP will create a major transformation in the provision of medical care.
Those not familiar with the functionality of the AVP are referred to the multiple video sources that describe its operation. Here, we also provide video documentation of hypothetical scenarios in clinical care. Before describing the case uses of AVP in medicine, it is important to make a distinction that fundamentally differentiates AVP-like products from virtual reality products. The AVP is best described as a product of “immersive layering” (IL) – that is IL builds layers of information that are added to the direct visual inspection the user has of reality.
AVP allows the user to directly observe its surroundings in a feature called “passthrough”. The AVP does this instantaneously with great fidelity and minimal granularity. With further product development, the quality of the passthrough will be indistinguishable from reality. The quality of passthrough seems sufficient to believe it doesn’t interfere with direct visualization needed in medicine (even for procedures). For example, users can play fast-reflex sports like ping-pong while using an AVP
.
While these products can be a tool for virtual reality, most of our descriptions will focus on the clinical applications and potential consequences of IL. It is important to note that the potential uses below are agnostic to the field of endeavor. Rather, AVP and IL serve to create a unique immersive environment that is best described as a digital assistant to its user.
Eye contact and limitations
The AVP precludes those in front of the user to observe the upper portion of the user’s face directly. While the AVP outside screen projects a view of the user’s face, this image is based on the user’s avatar (a.k.a., Persona), and direct eye contact and tracking are limited. Given the intrinsic and essential component of non-verbal communication skills for human connectivity in medicine, this will remain, for the time being, a potential limitation of AVP. Future product iterations might bring us products that minimize interference, such as transparent screens. Our description below of proposed uses assumes that such interference is existent and suboptimal for patient interaction. EyeSight is a built-in feature of the AVP that is supposed to project externally the eyes of its user. Still, we believe, at this time, the images are insufficiently realistic to be layered between the provider and patients.
Legality and services
Any product such as AVP will have to be compliant with all current regulations, including the Health Insurance Portability and Accountability Act (HIPAA). At the time of this writing, the product does not contain an API allowing for secure (enterprise) network management of protected health information. Once such options exist, much like other mobile devices like phones and tablets, PHI-enabled information will flow through the processing of AVP. After all, the AVP is merely “an iPhone on steroids!” and contains login security options. It is unclear how proposed solutions can become commercially viable- via intellectual property or as competition by provision of services.
Below are hypothetical scenarios where AVP could help.
Hospital care
Augmentation of clinical care – the AVP rounding team
AVP can enhance hospital clinical care. An obvious use could be to have a member of the clinical team use the device during rounds, where such a person remains a mobile source of critical clinical information. The rest of the team interacts normally with the patient while the designated AVP user complements the interactions by reactively providing data of interest to the clinical teams or going beyond by interjection with clinical information during clinical care discussion.
How could this information transfer happen optimally? A member using AVP could verbally convey this information, but this person could also select what to project over connected screens above a patient’s bed (via screen mirroring). Clinical devices could be connected to topographical locations (e.g., room number) that create automatic links between the patient’s EMR and their team – automatically populating the options for the credentialed user. Such a team approach will protect the human connectivity needed for trust and yet provide clinical decision-makers with optimal and relevant information.
Now imagine if this also occurs in the context of artificial intelligence, where pre-populated virtual screens convey critical information, anticipate risk, and suggest actions. To be effective, the connectivity ideally occurs automatically, based on location (e.g., 5G enabled, RFID). Users could also walk past patient rooms where critical information, including identifiers, is displayed on the outside wall of the patient room.
Nursing
Nursing care could also be improved by environmental monitoring of those activities inside the hospital room. The interactions will evolve from asking patients for identifiers, such as their birthdates, to validated biometrics that can more safely create these links. These biometrics could be streamlined to monitor medication administration, vital signs, progress with hydration and fluid exchange, etc. Inspection of a patient’s skin might be able to anticipate the risk of infection or pressure ulcers.
One important aspect to consider, as with all patient safety, is proper disinfection and sterilization of the AVP between users and users. Ensuring infection prevention and control, coupled with proper training, will allow for its safe use.
Experiential improvements
A very interesting aspect of AVP is that it can create “complete room” immersive experiences that go beyond the mere IL. For example, media preloaded in AVP allows the user to experience being in the middle of a recording room with a famous artist. A logical next step would be to consider how this could be enabled to allow remote visitors to feel in close proximity to a person who is hospitalized. Hospital rooms that are equipped with multiple cameras and microphones could allow a remote user with an AVP to have a high-fidelity experience of being present in such a room. The ability of someone who is unable to visit a loved one and yet has proximity to the real experience of being there is priceless. In return, the presence of that remote individual could also be reciprocated if the patient were to use a similar device. Even better, imagine other features, such as holographic projections of such a person, were visible to the patient.
The ability to hold what medical teams call “family conferences,” where all relevant stakeholders (local and remote) are present, would be advantageous by allowing better communication with patients and loved ones. These conferences are often held at critical junctures when important and oftentimes difficult clinical decisions need to be made. The ability to provide medical information, such as imaging findings or a complex surgical plan, to remote stakeholders is key to facilitating informed decision-making. The system could also provide an opportunity for improved hospital discharge planning, allowing key stakeholders to engage in this process and facilitate enhanced communication about discharge medications and follow-up care. One can simply imagine how simultaneous translation, close-captioning, or even automated sign languages could allow users with multiple backgrounds and capabilities to participate in seamless conversation, completely transforming the care experience through the provision of inclusive communication.
The features described above could also be available for patients now participating in “hospital-at-home” programs. Institutions like ours can provide hospital-level care to patients who remain in the comfort of their homes. Patients in a home hospital are equipped with connectivity devices that allow the medical teams to retain excellent connectivity to patient’s data. The current models only allow for transmitting information such as vital signs, weight, or patient-entered information. The availability of immersive remote monitoring could allow the medical teams to seek clues from the visual and auditory enhancements from diverse data sources.
Comfort approaches
Devices like the AVP could also assist in alleviating anxiety for those undergoing invasive procedures. Research that explores the soothing effects of images and sounds is needed. For instance, in the case of individuals with cognitive decline, the presence of loved ones during such procedures could be of great comfort. Lastly, other aspects, like navigating healthcare facilities via AVP-enhanced wayfinding, could make the already stressful experience easier for those undergoing care.
Outpatient uses
Frailty assessment
Many aspects of medical care that are considered important are often ignored because of time constraints. One such example is the determination of fitness (or the frailty status of patients). While many scales have been validated to allow better selection of fitness for certain therapies, the reality is in daily practice, and people make simplistic subjective assessments. Coupled with image analysis, evaluating the gait of a patient as they enter an office could be sufficient to provide objectivity. Nursing teams and physical therapists will then perform inspections to assess the risk of falls and fractures.
Dermatology
AVP would enhance routine care activities, such as the whole skin examination. A trained operator (perhaps not even a dermatologist) could conduct a detailed inspection with a secondary layer of evaluation being done by an expert dermatologist and with consequent behaviors (e.g., biopsy). The dermatologist evaluation could be done in less time with the augmentation provided by automated image analysis. AVP could significantly facilitate dermatologic evaluation of skin with different tonalities.
Phlebotomy
Multiple other examples come to mind. An enhanced ability for phlebotomists and nursing teams to insert intravenous cannulas with the assistance of AVR is likely. For the patient, virtual reality could serve to alleviate any anxiety that might accompany the procedure.
Mental health
Patients could use AVP for assistance with the care of individuals with dementia, establishing enhanced communications using images (real and virtual) of loved ones (alive and not), to allay anxiety and provide familiarity.
An argument has been made that, among other things, substance abuse disorders and mental health impairment is driven by existential anxiety and loneliness. We would not dare to propose that AVR could solve all these problems, but it could enhance connectivity for those suffering from loneliness. While some postulate that devices like AVP might distance and segregate humans, a more optimistic assessment might propose the opposite.
Diagnostic Radiology
The quality of passthrough images could allow radiologists to simultaneously read images while receiving artificial intelligence-enhanced feedback and interpretation of information. The enactment of 3D reconstruction of images would greatly enhance the diagnostic ability and certainty of radiologists. This could be derived not only from the topographical integration of multiple images, but also assisted by enhancements such as coloring and rotational visualization.
Many more things could be addressed in the outpatient area. But as the boundaries between what is considered care in need of hospitalization and care that can be provided remotely continue to blur, this analysis becomes more relevant. With creativity, medical teams should be unencumbered by many of the menial tasks that plague medicine and can lead to burnout. A physician with extraordinary memory or analytical ability will be less valuable than one who can enhance the human connectivity related to care.
Education
Healthcare providers
Devices such as AVP open the door for experiential learning to improve medical education. The ability to teach procedures to learners without practicing on live human beings instantly lowers overall risk. In distinction to virtual reality, immersive layers could provide additional context and information to learners as they interact with patients. These patients could be real-world patients or simulations. Providing three-dimensional anatomical models to medical students will allow for more sustained cognitive improvements. Teaching complex concepts, such as the neurological physical examination, will be done via the AVP.
Patients
The ability of AVP to create education modules for patients will enhance the clinicians’ ability to explain medical diagnoses. Such experiences could be created to explain the various aspects of a disease, including the diagnostic process, prognosis, treatment options, and actual physical description of treatments that involve procedures.
Procedures
Perhaps one of the most exciting and tantalizing potential uses for the AVP would be its integration in the procedural and surgical space. There are multiple clinical scenarios where an operator handles an instrument, and simultaneously must be looking at the monitor that provides visual feedback. These monitors are positioned in a way that maximizes convenience to the surgeon or the proceduralist. However, this experience is bound to be improved by the layering of similar screens into the direct clinical field, and subject to the control (even verbally) by the proceduralist. Systems such as the DaVinci™ robot could position a scrubbed surgeon in the direct surgical field, maybe even allowing ongoing exchange of instruments. An image or a video appears, and the AVP can be easily mirrored to other monitors in the room, allowing the team to understand occurrences. This IL should provide an operator with greater ease. This is the simple and obvious part: what follows?
Potentially a world in which the operator of a procedure receives real-time feedback layered over the procedural area and that provides enhanced information. Imagine data that is being analyzed in real-time via edge computing that provides feedback to an individual conducting a colonoscopy. That person could then react to this information and revisit areas of pathology missed, such as a flat polyp. Imagine if a neurosurgeon had functional mapping overlaying the resection of a glioblastoma. Imagine if an ultra-sonographer or an echocardiographer could actually see the images beneath the skin being captured. The possibilities are endless.
Enhancing Maintenance Efficiency and Patient Care in High Specialty Practices:
Behind the seamless delivery of radiotherapy medicine lies a complex equipment infrastructure, often requiring meticulous maintenance and significant financial investment in service contracts. When these systems encounter malfunctions, the repercussions are profound. Complex engineering requires painstakingly diagnosing step by step the root cause, and current methods of such troubleshooting are waiting to be augmented through the use of technology. The industry is now turning to digital twining—a technology that creates virtual replicas of physical assets. By integrating digital twinning with immersive technology such as Apple Vision Pro, a spacial computing transformative approach to maintenance emerges. Central teams comprising experts from equipment manufacturers can remotely guide onsite personnel through troubleshooting procedures, leveraging augmented reality overlays provided by the glasses. This collaborative approach minimizes downtime and ensures timely delivery of radiation treatments to patients.
Quality
It should be evident from the above discussion there are multiple ways by which IL can greatly increase the quality of medical care. Simple tasks, such as correctly identifying patients, including those with possible name confusion, will be averted. A procedural pause before surgery could also identify this site of a procedure to avoid “wrong side” surgery. Our ability to anticipate iatrogenic complications will also be enhanced. Prevention of falls and ulcers will improve.
Medical research
In this article, we focus less on the impact of this device on medical research. However, many uses are apparent. The association of new models of molecular structures and their interactions with ligands is an obvious one. Another interesting one, unrelated to recent developments, would be for principal investigators and editors to proofread materials to be published to ensure accuracy. Automated image analysis could prevent the publication of erroneous or altered images. The discussion of such uses is beyond the scope of this manuscript.
Conclusion
As Marie Curie once said, “things should not be feared, rather, they should be understood.” The advent of tools such as AVP will transform the provision of clinical care in ways that we cannot yet fully comprehend. The possibility to improve the quality of care, including better outcomes, and with an improved experience for both patients and providers is palpable. Some of the proposed uses will need canonical clinical validation, while others will be self-evident as beneficial.
The main question is, how long will it take for this to be implemented?
[1] https://www.apple.com/newsroom/2024/02/apple-announces-more-than-600-new-apps-built-for-apple-vision-pro/
Keywords: Apple Vision Pro™, #spatialcomputing, #VirtualReality, #Mobilecomputing, #VR, #AR, #Apple