Linear Factor Models

Linear Factor Models is when we have data $\{{\mathbf{x}}^{(n)}{\}}_{n=1}^N$, we extract the latent representation ${\mathbf{h}}^{(n)}({\mathbf{x}}^{(n)})$ and then we can reconstruct the original data in a linear map $W$:

$${\mathbf{x}}_n=W{\mathbf{h}}_n+[\dots ]$$

• It is used for unsupervised learning.
• $\mathbf{h}$ is an (unobserved) abstraction, i.e. latent features of the data. Different LFMs are used to learn different types of datasets:

1. Probabilistic Principle Component Analysis
   • Underlying factors ${\mathbf{h}}_n$ are distributed normally
   • Source components $h_i$s are independent of each other
2. Independent Component Analysis
   • Underlying factors ${\mathbf{h}}_n$ are not distributed normally
   • Source components $h_i$s are independent of each other
3. Slow Feature Analysis
   • Underlying data is in time series $\mathbf{x}(t)$
   • The abstract features are “slow-moving”
4. Sparse Coding and Dictionary Learning
   • Underlying data has a certain ‘vocabulary’ or dictionary $\mathbf{D}$
   • Data can be reconstructed by combinding just a few of the ‘vocabulary’.