Probabilistic Principle Component Analysis

Motivation. Principle Component Analysis was about transforming data into the explanatory variables (=principle components). PPCA is mathematically similar, but it thinks of each source variable as a random variable.

def. Probabilitic Principle Component Analysis (PPCA) model. We have observed data $\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_N\}$ with ${\mathbf{x}}_n\in {\mathbb{R}}^{\text{dim}({\mathbf{x}}_{\mathbf{n}})}$ and we want to extract latent features into $\mathbf{h}\in {\mathbb{R}}^{\text{dim}(\mathbf{h})}$. Naturally, $\text{dim}({\mathbf{x}}_n)=d\gg q=\text{dim}(\mathbf{h})$.

$${\mathbf{x}}_n=W{\mathbf{h}}_n+\mu +{ϵ}_n$$

where

• ${\mathbf{h}}_n\sim \mathcal{N}(0,I_d)$ i.e. multivariate standard normal distribution
• ${ϵ}_n\mathcal{N}(0,{\sigma }^2{I}_d)$ noise, i.e. multivarite normal distribution with mean $0$, variance ${\sigma }^2$ for each
• $W\in {\mathbb{R}}^{\text{dim}({\mathbf{x}}_n)\times \text{dim}({\mathbf{h}}_n)}$ Observe that then ${\mathbf{x}}_n\sim \mathcal{N}(\mu ,\mathbf{C})$ where $\mathbf{C}=W{W}^{\top }+{\sigma }^2{I}_d$ Objective is to find the optimal $W$, noise variance ${\sigma }^2$ and $\mu$ given observed data. We use MLEs for this. Steps are briefly outlined below. Objective function $f$ is the log-likelihood:

$$f=\ln \mathcal{L}=-\frac{N\cdot \text{dim}({\mathbf{x}}_n)}{2}\log (2\pi )-\frac{N}{2}\log |\mathbf{C}|-\frac{1}{2}\sum _{n=1}^N({\mathbf{x}}_n-\mathbf{\mu }{)}^{\top }{\mathbf{C}}^{-1}({\mathbf{x}}_n-\mathbf{\mu })$$

The analytic solution using eigendecompotision is:

$$\begin{array}{rl}{\mathbf{\mu }}_{\text{MLE}}& =\frac{1}{N}\sum _{n=1}^N{\mathbf{x}}_n\\ {\sigma }_{\text{MLE}}^2& =\frac{1}{d-q}\sum _{i=q+1}^d{\lambda }_i\\ {\mathbf{W}}_{\text{MLE}}& ={\mathbf{U}}_q({\mathbf{\Lambda }}_q-{\sigma }_{\text{MLE}}^2{\mathbf{I}}_q{)}^{1/2}\mathbf{R}\end{array}$$

where

• ${\mathbf{\Lambda }}_q=\text{diag}({\lambda }_1,\dots ,{\lambda }_q)$ in decreasing order, is from a spectral decomposition of sample covariance $\frac{1}{N}{\sum }_{n=1}^N({\mathbf{x}}_n-{\mu }_{\text{MLE}})({\mathbf{x}}_n-{\mu }_{\text{MLE}}{)}^{\top }=U\mathbf{\Lambda }{U}^{\top }$
• $\mathbf{R}\in {\mathbb{R}}^{q\times q}$ is an arbitrary orthogonal rotation matrix