Restricted Boltzmann Machines

def. Restricted Boltzmann Machine.¹ Let a partitioned neural network with visible layer $\mathbf{x}$ and hidden (=latent) layer $\mathbf{h}$, with weight matrix $W$ and biases $\mathbf{c}$ for visible layer and $\mathbf{b}$ for hidden layer. Then we can define a probability distribution over the whole network:

$$p(\mathbf{x},\mathbf{h})=\frac{\exp (-E(\mathbf{x},\mathbf{h}))}{Z}$$

where $E(\mathbf{x},\mathbf{h})$ is called the “Energy function”

$$\begin{array}{rl}E(\mathbf{x},\mathbf{h})& =-{\mathbf{h}}^{\top }\mathbf{Wx}-\underset{\text{ data bais }}{\underbrace{{\mathbf{c}}^{\top }}}\mathbf{x}-\underset{\text{ latent bias }}{\underbrace{{\mathbf{b}}^{\top }}}\mathbf{h}\end{array}$$

In most cases, each $x_i,h_j\in \{0,1\}$ (Bernouilli–Bernouilli) or $x_i\in \mathbb{R},h_j\in \{0,1\}$ (Gaussian–Bernouilli) Conditional Inference. Given data ${\mathbf{x}}^{(\text{realized})}$ we obtain the distribution of hidden layer

$$\begin{array}{rl}& \\ p(\mathbf{h}|{\mathbf{x}}^{\text{realized}})& =\prod _{j}p(h_j|{\mathbf{x}}^{\text{realized}})\\ p(h_j=1|{\mathbf{x}}^{\text{realized}})& =\sigma (b_j+\underset{\text{ j-th row }}{\underbrace{W_{j,\cdot }}}\mathbf{x})\end{array}$$

from which we can “sample” in the latent distribution to get ${\mathbf{h}}^{\text{realized}}$. Alternatively, given a latent representation ${\mathbf{h}}^{\text{realized}}$ we obtain the distribution of generated “data”:

$$\begin{array}{rl}p(\mathbf{x}|{\mathbf{h}}^{\text{realized}})& =\prod _{k}p(x_k|{\mathbf{h}}^{\text{realized}})\\ p(x_j=1|{\mathbf{h}}^{\text{realized}})& =\sigma (c_k+{\mathbf{h}}^{\top }\underset{\text{ k-th col }}{\underbrace{W_{\cdot ,k}}})\end{array}$$

Training. We minimize the log-likelihood to model the data distribution $p(\mathbf{x})$ to fit the data best:

$$\underset{W,\mathbf{c},\mathbf{b}}{\min }\frac{1}{T}\sum _{\forall t}-\ln p(x^{(t)})$$

Footnotes §

1. Neural networks [5.1] : Restricted Boltzmann machine - definition - YouTube ← this whole playlist is very useful. ↩