טכניון מכון טכנולוגי לישראל
הטכניון מכון טכנולוגי לישראל - בית הספר ללימודי מוסמכים  
Ph.D Thesis
Ph.D StudentRyder Darian
SubjectThe Effect of Auditory and Visual Emotional Stimuli in VR
on Episodic Memory
DepartmentDepartment of Education in Science and Technology
Supervisor Professor Miriam Reiner
Full Thesis textFull thesis text - English Version


Abstract

Memory is an existentially essential need for daily functioning. It enables us to remember things from our prior knowledge, associate them to our present situation and information and make sense of them according to what we previously learned by contextually encoding them. Episodic memory (EM), aka “flashbulb memory”, enables us remembering instantaneously, through a single event, with rich, vivid details. It is enhanced by emotionally salient features through multi-modal sensory integration. Following the exponentially increasing amount of knowledge required in our information overloaded, evolving world - EM seems an ideal candidate to improve memorization and shortening time-to-knowledge, as it increases remembering, support associativity, facilitate recognition followed by better schematization and recollection.

Our study aims to uncover the effects of emotional audio-visual stimuli on EM using three quantitative controlled empirical experiments through ecologically valid, immersive virtual reality (VR) environments and identify the differences between items memorized under varying emotional conditions.

For these experiments, we have selected musical pieces heard during task performance as the auditory stimuli and color hues emitted from objects in the VR environment as the visual stimuli. These stimuli were shown to have an emotional effect on EM and subsequently, attention and accuracy. Positive and Negative auditory and visual stimuli were combined into Sensory Stimuli Sets (S3).

Participants performed memory encoding and cued recall trials (immediate and delayed) under different S3 conditions to evaluate episodic memorization. In the first experiment participants experienced two audio-visual modes of object sequence presentations to validate the VR construct. The second experiment explored the effect of disparate color hues on object sequence memorization. The third experiment, combined and enhanced the previous two, evaluating various S3 while increasing realism and user experience. A Rey Auditory Verbal Learning Test (RAVLT) was administered for additional validation.

Our results show episodic memorization improvement under positive S3 and the contrary for negative S3 with moderate to high effect sizes on scores across trials (p<0.003, η2=0.569; 0.725; 0.503 trial 1 to 3 respectively). Experiment scores bear convergent validity with the RAVLT Retention and Serial Recall scores with high correlation for both (p<0.05, R2 =0.843).

Our findings support our assumption regarding the salience of positive/negative audio-visual stimuli on episodic memorization. This is consistent with studies showing episodic memory emotional enhancement (EEM) of similar stimuli. EEM occurs through diverse activation patterns of the Hippocampus. These patterns enhance attention during encoding, perform emotional synaptic tagging during consolidation, immediately enhanced EM plasticity, enhance recollection and later merging of audio-visual information (Amygdala, Medial Temporal Lobe, Superior Temporal Gyrus (STG) and medial frontal gyrus respectively).

Together, VR emotional stimuli augment sequence memorization through known emotion-cognition patterns, affecting cognitive performance, reducing impulsivity and increasing attention and accuracy, convergent with RAVLT test scores. We conclude by analyzing some of our study’s limitations, most notably the relatively small number of participants and suggest a follow-up study for results substantiation. We expect gaining significant results for near-significant trends and extend the breadth of our cognitive enhancement infrastructure with additional capabilities, for eventual integration in both psychometric and educational infrastructures.