M.Sc Thesis

M.Sc StudentBarzelay Zohar
SubjectRelating Audio and Video of Multiple Simultaneous Events
DepartmentDepartment of Electrical and Computer Engineering
Supervisor PROF. Yoav Schechner
Full Thesis textFull thesis text - English Version


Cross-modal analysis offers information beyond that extracted from individual modalities. Consider a camcorder having a single microphone in a cocktail-party: it captures several moving visual objects which emit sounds. A task for audio-visual analysis is to identify the number of independent audio-associated visual objects (AVOs), pinpoint the AVOs' spatial locations in the video and isolate each corresponding audio component. Part of these problems were considered by prior studies, which were limited to simple cases, e.g., a single AVO or stationary sounds. We describe an approach that seeks to overcome these challenges. The approach does not inspect the low-level data. Rather, it acknowledges the importance of mid-level features in each modality, which are based on significant temporal changes in each modality. A probabilistic formalism identifies temporal coincidences between these features, yielding cross-modal association and visual localization. This association is further utilized in order to isolate sounds that correspond to each of the localized visual features. This is of particular benefit in harmonic sounds, as it enables subsequent isolation of each audio source, without incorporating prior knowledge about the sources. We demonstrate this approach in challenging experiments. In these experiments, multiple objects move simultaneously, creating motion distractions for one another, and produce simultaneous sounds which mix. Yet, the results demonstrate spatial localization of correct visual features out of hundreds of possible candidates, and isolation of the non-stationary sounds that correspond to these distinct visual features.