Probabilistic Generative Models

Motivation. Inspiration is from Maximum Likelihood Estimators. Consider classifying into two classes, ${\mathcal{C}}_1,{\mathcal{C}}_2$ If we consider $\mathbf{w}$, the weights as a parameter is a probability distribution for class ${\mathcal{C}}_1$, classification output as such:

• $p({\mathcal{C}}_1\mid \mathbf{x})$: given datapoint $\mathbf{x}$ the probability (distribution) of it being in class $1$. The pdf of this is what we want to find.
• $p(\mathbf{x}\mid {\mathcal{C}}_1)$: given that data is in class $1$, what is the distribution of the datapoints? We know this from the data. We will construct the former from the latter.

From $\mathbf{x}$ as a multivariate Normal Distribution §

If we have $\mathbf{x}$ as a multivariate normal distribution, it has the probability distribution:

$$p(\mathbf{x}\mid {\mathcal{C}}_1)=\frac{1}{(2\pi {)}^{\frac{d}{2}}|\Sigma {|}^{\frac{1}{2}}}{e}^{-\frac{1}{2}(X-{\mu }_1{)}^T{\Sigma }^{-1}(X-{\mu }_1)}$$

where

• $\overrightarrow{{\mu }_1}$ is the mean point of datapoints in class $1$
• $\Sigma$ is the Covariance matrix for this distribution Both off these we do not know yet. Then using Bayes’ Rule:

$$\begin{array}{rlr}p({\mathcal{C}}_1\mid \mathbf{x})& =\frac{p(x|C_1)\cdot ({\mathcal{C}}_1)}{p(x|C_1)p(C_1)+p(x|C_2)p(C2)}& \text{by Bayes’}\\ & =\frac{\exp [\stackrel{\text{ let ak }}{\overbrace{\ln p(x|{\mathcal{C}}_k)p({\mathcal{C}}_k)}}]}{{\sum }_{\forall j}\exp [\underset{\text{ let aj }}{\underbrace{\ln p(x|{\mathcal{C}}_j)p({\mathcal{C}}_j)}}]}& \text{add exp \& log for sigmoid}\\ & =\sigma (a_k|a_1,\dots ,a_n)& \text{by def of sigmoid}\end{array}$$

where

• See ^unj2yy for the definition
• Softmax ensures this is a probability distribution. Now, we calculate $a_k$. Substitute the multivarate normal pdf into $a_k$:

$$\begin{array}{rlr}{a}_k& =\ln p(x|{\mathcal{C}}_k)p({\mathcal{C}}_k)& \\ & =\underset{\text{ let C }}{\underbrace{\ln \frac{1}{(2\pi {)}^{D/2}}+\ln \frac{1}{|\Sigma {|}^{1/2}}}}-\frac{1}{2}({\mathbf{x}}^{\top }-{\mu }_k^{\top }){\Sigma }^{-1}(\mathbf{x}-{\mu }_{\mathbf{k}})+\ln p({\mathcal{C}}_k)& \\ & =C-\frac{1}{2}{\mathbf{x}}^{\top }{\Sigma }^{-1}\mathbf{x}+\frac{1}{2}{\mathbf{x}}^{\top }{\Sigma }^{-1}{\mu }_k+\frac{1}{2}{\mu }_k^{\top }{\Sigma }^{-1}\mathbf{x}\underset{\text{ let wk0 }}{\underbrace{-\frac{1}{2}{\mu }_k^{\top }{\Sigma }^{-1}{\mu }_k+\ln p({\mathcal{C}}_k)}}& \\ & =C-\frac{1}{2}{\mathbf{x}}^{\top }{\Sigma }^{-1}\mathbf{x}+\underset{\text{ let wk }}{\underbrace{{\mu }_k^{\top }{\Sigma }^{-1}}}\mathbf{x}+w_{k_0}& \\ & ={{\mathbf{w}}_{\mathbf{k}}}^{\top }\mathbf{x}+w_{k,0}-\frac{1}{2}{\mathbf{x}}^{\top }{\Sigma }^{-1}\mathbf{x}+C& {\Sigma }^{-1}\text{ is symtrc}\end{array}$$

Finally, we plug this into our softmax function to get:

$$\begin{array}{rl}p({\mathcal{C}}_k|\mathbf{x})& =\frac{\exp \left({{\mathbf{w}}_{\mathbf{k}}}^{\top }\mathbf{x}+w_{k,0}\cancel{-\frac{1}{2}{\mathbf{x}}^{\top }{\Sigma }^{-1}\mathbf{x}+C}\right)}{{\sum }_{\forall j}\exp \left({{\mathbf{w}}_{\mathbf{j}}}^{\top }\mathbf{x}+w_{j,0}\cancel{-\frac{1}{2}{\mathbf{x}}^{\top }{\Sigma }^{-1}\mathbf{x}+C}\right)}\\ & =\frac{\exp \left({{\mathbf{w}}_{\mathbf{k}}}^{\top }\mathbf{x}+w_{k,0}\right)}{{\sum }_{\forall j}\exp ({\mathbf{w}}_{\mathbf{j}}^{\top }\mathbf{x}+w_{j,0})}\\ & \end{array}$$

Finding the optimal parameters $\mathbf{w}$ §