Affective Computing and Multimodal Understanding

Description:

This effort in our lab focuses on modeling internal affective and cognitive states such as emotion, familiarity, and spontaneous thought by extracting meaningful features from language, eye gaze, and other behavioral signals. We use machine learning to link these surface cues to underlying experiences, enabling more adaptive and emotionally aware AI systems.

Our work spans multiple contexts: we analyze linguistic patterns in descriptions of spontaneous thoughts to differentiate déjà vu, autobiographical memories, and unexpected ideas; we study retrospective verbalizations in collaborative tasks to reveal how people express internal states like frustration or confidence; and we use eye-gaze data in virtual reality to detect subtle feelings of familiarity. These studies showcase how multimodal signals can be harnessed to infer internal states, bridging cognitive science and machine learning in applied settings.

Publications:

Venkatesha, V., Poulos, M. C., Steadman, C., Mills, C., Cleary, A. M., and Blanchard, N. (2025). A Linguistic Analysis of Spontaneous Thoughts: Investigating Experiences of Deja Vu, Unexpected Thoughts, and Involuntary Autobiographical Memories. In: Proceedings of the Annual Meeting of the Cognitive Science Society, 47, pp. 1720-1726.

Anindho, S., Venkatesha, V., and Blanchard, N. (2025). A Methodological Framework for Capturing Cognitive-Affective States in Collaborative Learning. In: Educational Data Mining (EDM) Interactive Workshop: Multimodal, Multiparty Learning Analytics (MMLA).

Anindho, S., Venkatesha, V., Bradford, M., Cleary, A. M., and Blanchard, N. (2025). An Exploration of Internal States in Collaborative Problem Solving. In: Kurosu, M., Hashizume, A. (Eds.) Human-Computer Interaction. HCII 2025. Lecture Notes in Computer Science, vol 15768. Springer, Cham., pp. 135–150.

CHARTIER, T., CASTILLON, I., VENKATESHA, V., CLEARY, A. M., BLANCHARD, N. T., ORCiD, I. D., & Chartier, T. Using Eye Gaze to Differentiate Internal Feelings of Familiarity in Virtual Reality Environments: Challenges and Opportunities. Annual Review of Cybertherapy And Telemedicine 2024, 96.

Castillon, I., Venkatesha, V., VanderHoeven, H., Bradford, M., Krishnaswamy, N., & Blanchard, N. (2022). Multimodal features for group dynamic-aware agents. In Interdisciplinary Approaches to Getting AI Experts and Education Stakeholders Talking Workshop at AIEd. International AIEd Society.

Steadman, C., Venkatesha, V., Poulos, C., Cleary, A.M., Blanchard, N., Mills, C. Involuntary Thoughts in Older vs. Younger Adults: A Multidisciplinary Approach to Investigating D´ej`a Vu, Involuntary Au- tobiographical Memories, and Unexpected Thoughts. Technology, Mind, and Behavior Journal Special Issue. .