RFM-HRI: A Multimodal Dataset of Medical Robot Failure, User Reaction, and Recovery Preferences for Item Retrieval Tasks

A multimodal dataset capturing 214 human-robot interactions from 41 laypersons and healthcare workers responding to crash-cart robot failures in an item-retrieval task. We developed an annotation framework using facial action units (OpenFace) and speech transcription (Whisper) to quantify user reactions and preferred recovery modes in embodied AI. User responses to robot failures vary significantly by context, ranging from confusion in early trials to frustration in later ones, and preferred recovery strategies combine verbal correction with transparency cues. User emotions were tracked with the Self-Assessment Manikin (SAM) to record valence and perceived control.