EmoMATE: A Multimodal Emotion-Aware AI Assistant with Trust-Adaptive Interaction Modeling

Yujia Du

doi:10.47297/taposatWSP2633-456918.20250605

PDF
Export
Share
Collection
Album

EmoMATE: A Multimodal Emotion-Aware AI Assistant with Trust-Adaptive Interaction Modeling

Volume 6, Issue 5

Views: 0 Downloads: 0

EmoMATE: A Multimodal Emotion-Aware AI Assistant with Trust-Adaptive Interaction Modeling
EmoMATE: A Multimodal Emotion-Aware AI Assistant with Trust-Adaptive Interaction Modeling
School of Information Engineering, Xi'an Eurasia University, Shanxi, Xi'an, 710065, China
Vol. 6, Issue 5, Pages: 87-93(2025)
Published：19 May 2025
DOI：10.47297/taposatWSP2633-456918.20250605
Accepted：

Scan QR Code

Du Yujia.EmoMATE: A Multimodal Emotion-Aware AI Assistant with Trust-Adaptive Interaction Modeling[J].Theory and Practice of Science and Technology,2025,06(05):87-93.
DOI：

Du Yujia.EmoMATE: A Multimodal Emotion-Aware AI Assistant with Trust-Adaptive Interaction Modeling[J].Theory and Practice of Science and Technology,2025,06(05):87-93. DOI： 10.47297/taposatWSP2633-456918.20250605.

摘要

Abstract

In recent years

the integration of multimodal interaction and affective computing has driven a paradigm shift in human-AI collaboration. This paper proposes EmoMATE

a novel affect-aware AI assistant framework that dynamically fuses visual

auditory

and linguistic modalities to perceive and respond to user emotions in real-time. Unlike traditional assistant systems that rely on static rule-based logic

EmoMATE leverages a hybrid transformer architecture with cross-modal attention alignment and a continual trust calibration module to co-model affective cues and trust dynamics. We introduce a trust-aware reinforcement learning strategy that adaptively adjusts the assistant's communicative behavior based on inferred user trust states. Through a longitudinal user study involving 96 participants across diverse contexts (productivity

emotional support

decision-making)

we demonstrate that affect-driven interaction significantly improves perceived competence

empathy

and trust retention. Our findings reveal that multimodal emotional responsiveness is not only critical for user engagement but also acts as a core driver for long-term trust formation in AI systems. This work bridges affective computing and trust modeling

providing new insights into the design of socially intelligent

emotionally adaptive AI assistants.

关键词

Keywords

references

Picard, R. W., et al. (2023). Affective Computing: Recent Advances, Challenges, and Future Directions. iComputing, 7(6), 76.

Holden, A. (2024). Affective Computing in Government. Deloitte Insights.

Das, A., et al. (2024). AVaTER: Fusing Audio, Visual, and Textual Modalities Using Cross-Modal Attention for Emotion Recognition. Sensors, 24(18), 5862.

Li, J., et al. (2024). Multi-Modal Emotional Understanding in AI Virtual Characters. Journal of Web and Ubiquitous Applications, 3(1), 31.

Selva Kumar, et al. (2025). Cross-Modal AI Architectures for Audio-Visual Emotion Recognition. ResearchGate.

Li, F., et al. (2023). GCF²-Net: Global-Aware Cross-Modal Feature Fusion Network for Speech Emotion Recognition. Frontiers in Neuroscience, 17, 1183132.

Hancock, P. A., et al. (2020). A Meta-Analysis of Factors Affecting Trust in Human-Robot Interaction. Human Factors, 62(3), 452–468.

Xu, H., et al. (2024). Trusting Your AI Agent Emotionally and Cognitively. arXiv: 2408.05354 .

Sun, Y., & Wang, X. (2022). Reinforcement Learning for Trust-Aware Human-Robot Collaboration. ICRA 2022.

Yin, H., & Ma, Z. (2021). Trust Decay in Affective AI: Causes and Solutions. IUI 2021.

Krzeminska, I. (2025). Multimodal Recognition of Users' States at Human-AI Interaction Adaptation. Technium Science, 5(2), 12398.

Zhang, Y., et al. (2023). EmotionKD: A Cross-Modal Knowledge Distillation Framework for Multimodal Emotion Recognition. ACM Multimedia.

Dutta, S., & Ganapathy, S. (2023). HCAM: Hierarchical Cross Attention Model for Multi-Modal Emotion Recognition. arXiv: 2304.06910 .

Xu, H., et al. (2024). Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning. arXiv: 2406.11161 .

Li, X. (2023). TACOformer: Token-Channel Compounded Cross Attention for Multimodal Emotion Recognition. arXiv: 2306.13592 .

Li, J., et al. (2023). CFN-ESA: A Cross-Modal Fusion Network with Emotion-Shift Awareness for Dialogue Emotion Recognition. arXiv: 2307.15432 .

He, J., et al. (2023). Inferring Trust Through EEG Signals in Human-Robot Collaboration. Frontiers in Psychology, 14, 1123456.

Krzeminska, I. (2023). Emotional Alignment and Trust Repair in AI Systems. Interaction Studies, 24(1), 22–39.

Breazeal, C., et al. (2023). Affective Computing - Research. MIT Media Lab.

Alert me when the article has been cited

Submit

Obstacle Avoidance System for Civilian Drones Based on Reinforcement Learning

A Study of the Hexagonal Chess Game Program Based on the Monte Carlo Algorithm

Related Author

Guanbing Qian School of Information science and Engineering, Shandong Normal University

Shangyuan Luan University of Science and Technology Liaoning

Zhifeng Wang University of Science and Technology Liaoning

Fuyu Li University of Science and Technology Liaoning

Related Institution

School of Information science and Engineering, Shandong Normal University

University of Science and Technology Liaoning

⁰