Independent Component Analysis

Motivation. Consider the problem where $q$ speakers are playing separate songs, and $d$ mics are picking them up. Just from the $d$ mic’s signals, can you separate the songs playing at each speaker? (=Blind Source Separation problem)

def. Independent Component Analysis (ICA) Model. We have observed data $\{{\mathbf{x}}_n\in {\mathbb{R}}^d{\}}_{n=1}^N$ and we want to extract ‘sources’ $\{{\mathbf{h}}_n\in {\mathbb{R}}^q{\}}_{n=1}^N$.

$${\mathbf{x}}_n=A{\mathbf{h}}_n$$

where

• $A$ is the hypothetical, unobserved ‘mixing matrix’.
• we assume any source components $h_i,h_j$ are independent of each other We want to find how to unmixing matrix $W$ which gets $\hat{\mathbf{h}}=W{\mathbf{x}}_n$. Intiution. For this, we take advantage of the central limit theorem—the (weighted) summation of any distribution will tend to a normal distribution. In this case, the mixing matrix does a weighted summation of each source component $h_i$, thus ${\mathbf{x}}_n$ will the ‘more normal’ than multivariate ${\mathbf{h}}_n$. Method 1. Maximization of Negentropy. Gaussian distribution has the largest entropy. Thus we can see how far the distribution of ${\mathbf{x}}_n$ is from gaussian and make it farther.

$$\underset{W}{\max }\frac{1}{N}\sum _{n=1}^{N}H[\mathcal{N}(0,\cdot )]-H[{\mathbf{x}}_n]$$

Method 2. Maximization of Kurtosis.