Massive multiple-input multiple-output (MIMO) communication is a promising technology for increasing spectral efficiency in wireless networks. Two of the main challenges massive MIMO systems face are the increase in hardware complexity due to the massive number of antennas and degraded channel estimation accuracy due to pilot contamination.

In this work we study these challenges from a signal processing point of view: First,

we treat the problem of reducing the hardware complexity by using fewer RF chains

than antennas at both the transmitter and receiver, and suggest a hybrid digital-analog processing scheme. We examine several different fully- and partially-connected architectures and design the beamformers to minimize the estimation error in the data. For the hybrid precoder, we introduce a framework for approximating the optimal fully-digital precoder with a feasible hybrid one. For combiner design, we exploit the structure of the MSE objective and develop a greedy ratio trace maximization technique, that achieves low MSE under various settings.

Next, we study the pilot contamination problem. We consider a multi-cell multiuser

scenario where the base stations use fewer RF chains than antennas via analog

combining and examine a statistical model in which the channel covariance obeys a

Kronecker structure. In this setting, we prove that the analog combiner design can

be performed independently of the pilot sequences. Given the resulting combiner, we

derive a closed-form expression for the optimal pilot sequences in the fully-separable

case and suggest a greedy sum of ratio traces maximization method for designing suboptimal pilots in the partially-separable scenario. We demonstrate the performance of our algorithms via simulations.