CS 675 HW2

Problem 1

Let $h_{3} = \overset{y}{^}$ for consistencey in notation. Assume vectors are column vectors. First we calculate:

\frac{\partial ℓ}{\partial h _{3, i}} \frac{\partial ℓ}{\partial h _{3}} = \frac{- 2}{N} (y_{i} - h_{3, i}) = - \frac{2}{N} (y - h_{3})^{⊤} = let δ_{3}^{⊤} shape 1 \times D_{3}

Then we consider the tree of computation as in class. Let $a_{k} = W_{k}^{⊤} h_{k - 1} + b_{k}$

$\frac{\partial h _{3}}{\partial W _{3}^{⊤}} = h_{2}$ to get $\frac{\partial ℓ}{\partial W _{3}^{⊤}} = h_{2} δ_{3}^{⊤}$ 🔴
$\frac{\partial h _{3}}{\partial b _{3}} = 1$ to get $\frac{\partial ℓ}{\partial b _{3}} = δ_{3}^{⊤}$ 🔴
$\frac{\partial h _{3}}{\partial h _{2}} = W_{3}$ to get $\frac{\partial ℓ}{\partial h _{2}} = W_{3} δ_{3}^{⊤} = l e t δ_{2}$ Next layer:
$\frac{\partial h _{2}}{\partial a _{2}} = \frac{\partial g ( \cdot )}{\partial ( \cdot )}$
$\frac{\partial a _{2}}{\partial h _{1}} = W_{2}^{⊤}$ to get $\frac{\partial ℓ}{\partial h _{1}} = W_{2}^{⊤} g^{'} (\cdot) δ_{2} = l e t δ_{1}$
$\frac{\partial h _{2}}{\partial W _{2}^{⊤}} = h_{1}$ to get $\frac{\partial ℓ}{\partial W _{2}^{⊤}} = h_{1} δ_{2}^{⊤}$ 🔴
$\frac{\partial h _{2}}{\partial b _{2}} = 1$ to get $\frac{\partial ℓ}{\partial b _{2}} = δ_{2}^{⊤}$ 🔴 Similarly:
get $\frac{\partial ℓ}{\partial W _{1}^{⊤}} = h_{0} δ_{1}^{⊤} = x δ_{1}^{⊤}$ 🔴
get $\frac{\partial ℓ}{\partial b _{1}} = δ_{1}^{⊤}$ 🔴

Results marked with 🔴