Slow Feature Analysis

Motivation. In a time series data, the latent features change more slowly than the actual data. For example in a video of a running zebra, the pixels of the video rapidly changes, but the ‘feature’ that there is a zebra in the frame does not change. We take advantage of this.

def. Slow Feature Analysis (SFA). We have continuous time series data $\mathbf{x}(t)$

1. We extract them into latent features over time $\mathbf{h}(t)$ via a set of non-linear functions $\mathbf{g}$.
   • These non-linear functions are somewhat arbitrarily chosen. It can simply be something like $x_i^2$ or $x_i+x_j$, etc.
2. Then we linearly combine these to get the output signal via matrix $W$

$$\mathbf{x}(t)\stackrel{\mathbf{g}}{⟶}\mathbf{h}(t)\stackrel{W}{⟶}\mathbf{y}(t)$$

Objectiv e. Our goal is to find $W$ such that $\mathbf{y}(t)$ changes very slowly. Thus the objective is to minimize:

$$\underset{W}{\min }{\mathbb{E}}_t[{\left(\frac{d}{dt}y(t)\right)}^2]\equiv \underset{W}{\min }\frac{1}{T}\stackrel{\text{ total rate of change }}{\overbrace{{\int }_0^{\infty }\underset{\text{ rate of change of y }}{\underbrace{y^{\prime }(t)}}dt}}$$

under constraints 3. Zero mean: ${\mathbb{E}}_t(y(t))=0$ ← because shifts doesn’t matter 4. Unit variance: ${\mathbb{E}}_t(y(t{)}^2)=1$ ← because amplitude doesn’t matter