|
|
|
Computer Science Department Colloquium
Toward Augmented Reality Guidance for High-Stakes Medical Care
Speaker:
Jonathan Segal, PhD Candidate, AIRLab Cornell Tech
When: 3:00PM ~ 4:00PM, Friday October 17, 2025
Where: CSB 130 
Abstract: Emergency departments demand rapid, coordinated action, including role assignment, time-critical interventions, and accurate dosage calculations under high pressure. Delays or mistakes often occur when clinicians must recall information, review reference materials, or communicate across the team. Augmented reality head-mounted displays (AR-HMDs) offer a potential solution by presenting relevant data such as timers, dosage calculations, or role labels directly in the user’s field of view. By delivering personalized, real-time information where and when it is needed, AR-HMDs reduce context switching, speed up decision-making, and help teams maintain synchronized workflows under pressure. To explore this potential, a participatory design study was conducted with 12 healthcare workers which produced Unity-based prototypes for role tags, timers, and dosage panels. These prototypes designed by participants highlighted opportunities such as hands-free spatial cues as well as challenges such as visual distraction and input ambiguity in applying AR-HMDs to critical care. A major outcome was that healthcare workers wanted only the most relevant information at any given time, which motivated the development of a procedural assistance system to address this need.
To fulfill this goal, we developed Generative Workflows (GenFlows) which is designed to fulfill that goal by transforming static protocols into customizable, deployable guidance. GenFlows uses visual-language models (VLMs) to extract procedural steps, a web interface for expert review and verification, and multi-device deployment. A systematic evaluation of three state-of-the-art VLMs, combined with interviews with 14 domain experts, showed that GenFlows enables clinicians to digitize and personalize workflows with greater efficiency. Together, these approaches illustrate how participatory design and generative AI can support safer and more contextually responsive guidance in high-stakes healthcare settings.
Bio: I'm currently working on my Ph.D. in Information Science at Cornell Tech and am a member of AIRLab led by Dr. Angelique Taylor. I am motivated by a future where augmented reality transforms healthcare delivery, particularly by assisting EMS and other medical professionals in high-stakes situations. Additionally, I am excited about many topics, such as particle physics, sports analytics, geospatial data, generative AI, and startups. I've earned a Bachelor's degree in Software Engineering at Iowa State, where I worked with Dr. Stephen Gilbert and Dr. Michael Dorneich at VRAC. I spent a summer at SLAC working with Dr. Jeff Shrager and Wan-Lin Hu. I've interned at BlackRock, Dwolla, and Corteva.
|