CS 675 HW5

Problem 2-1 §

(coding problems follow below) The energy function:

$$E(v,h;\theta )=-\left(\sum _i\sum _j\frac{W_{ij}{h}_j{v}_i}{{\sigma }_i}-\sum _i\frac{(v_i-b_i{)}^2}{2{\sigma }_i^2}+\sum _j{\alpha }_j{h}_j\right)$$

And the joint distribution $p(v,h)=\frac{\exp (-E(v,h))}{Z}$

$p(v_i=X|h)$ §

Isolate terms involving $v_i$:

$$\begin{array}{rl}E(v_i,h)& =-\left(\sum _j\frac{W_{ij}{h}_j{v}_i}{{\sigma }_i}-\frac{(v_i-b_i{)}^2}{2{\sigma }_i^2}\right)+\dots \\ & =v_i\left(\sum _j\frac{W_{ij}{h}_j}{{\sigma }_i}\right)-\frac{(v_i-b_i{)}^2}{2{\sigma }_i^2}\\ & =-\frac{(v_i-{\mu }_i{)}^2}{2{\sigma }_i^2}+\dots \end{array}$$

where

$${\mu }_i=b_i+{\sigma }_i\sum _j{W}_{ij}{h}_j.$$

And noting $Z$ as the regularization, $p(v_i|h)$is Gaussian:

$$p(v_i|h)=\frac{1}{\sqrt{2\pi {\sigma }_i^2}}\exp \left(-\frac{(v_i-{\mu }_i{)}^2}{2{\sigma }_i^2}\right).$$

$p(h_j=1|v)$ §

Isolate terms involving $h_j$:

$$E(v,h_j)=-\left(\sum _i\frac{W_{ij}{h}_j{v}_i}{{\sigma }_i}+{\alpha }_j{h}_j\right)+\dots$$

For $h_j\in \{0,1\}$:

$$\exp (-E(v,h_j))=\{\begin{array}{ll}\exp (0),& h_j=0\\ \exp \left({\sum }_i\frac{W_{ij}{v}_i}{{\sigma }_i}+{\alpha }_j\right),& h_j=1\end{array}.$$

Thus the conditional $p(h_j=1|v)$:

$$p(h_j=1|v)=\sigma \left(\sum _i\frac{W_{ij}{v}_i}{{\sigma }_i}+{\alpha }_j\right),$$

where $\sigma (x)=\frac{1}{1+\exp (-x)}$ is the sigmoid function.