M.Sc Thesis

M.Sc StudentKatz Ori
SubjectDifusion-Based Nonlinear Filtering for Multimodal
Data Fusion
DepartmentDepartment of Electrical and Computer Engineering
Supervisor ASSOCIATE PROF. Ronen Talmon
Full Thesis textFull thesis text - English Version


The problem of information fusion from multiple data-sets acquired by multimodal sensors has drawn significant research attention over the years. In this work, we focus on a particular problem setting consisting of a physical phenomenon or a system of interest observed by multiple sensors. We assume that all sensors measure some aspects of the system of interest with additional sensor-specific and irrelevant components. Our goal is to recover the variables relevant to the observed system and to filter out the nuisance effects of the sensor-specific variables.

We present an approach based on manifold learning, which is particularly suitable for problems with multiple modalities, since it aims to capture the intrinsic structure of the data and relies on minimal prior model knowledge. Specifically, we propose a nonlinear filtering scheme, which extracts the hidden sources of variability captured by two or more sensors, that are independent of the sensor-specific components.

In addition to presenting a theoretical analysis, we demonstrate our technique on real measured data for the purpose of sleep stage assessment based on multiple, multimodal sensor measurements. We show that without prior knowledge on the different modalities and on the measured system, our method gives rise to a data-driven representation that is well correlated with the underlying sleep process and is robust to noise and sensor-specific effects.

Another related topic that was motivated by the challenge of processing multiple modalities and is a fundamental element in any manifold learning technique is the ability to reveal the similarities between data points. In a series of recent studies, this was accomplished by the Mahalanhobis distance. Yet, the computation of the Mahalanhobis distance from data requires an estimation of the covariance matrix, which is challenging, especially when it is applied to high-dimensional data sampled from multi-scale stochastic dynamical systems.

Here, we further examine the computation of the Mahalanobis distance. Specifically, we address on the inherent tradeoff between preserving locality and minimizing the sample-variance error. We demonstrate the influence of the estimation on various aspects related to manifold learning and analyze the incurred errors. In addition, we present a new covariance matrix estimation method. Finally, we show the application of the proposed method to simulated data arising from a multiscale stochastic dynamical system and demonstrate its advantage.