Skip to main content

Derivative


Loss function

Mean Square Error

Loss=12i(yiaiL)2Loss = \dfrac{ 1 }{ 2 } \sum_i \left( y_i - a^L_i \right)^2 LossaiL=(yiaiL)\dfrac{ \partial Loss }{ \partial a^L_i } = - \left( y_i - a^L_i \right)

Binary Cross-Entropy

Loss=i[yilogaiL(1yi)log(1aiL)]Loss = \sum_i \left[ -y_i \log a^L_i - (1 - y_i) \log \left( 1 - a^L_i \right) \right] LossaiL=(yiaiL)aiL(1aiL)\dfrac{\partial Loss}{\partial a^L_i} = \dfrac{ - \left( y_i - a^L_i \right)}{a^L_i \left( 1 - a^L_i \right) }

Activation

logistic

ail=σ(zil)=logistic(zil)=11+ezila^l_i = \sigma ( z^l_i ) = logistic ( z^l_i ) = \dfrac{ 1 }{ 1 + e^{ -z^l_i } } σ(zil)zil=ezil(1+ezil)2=ail(1ail)\dfrac{\partial \sigma ( z^l_i )}{\partial z^l_i } = \dfrac{ e^{ -z^l_i } }{ \left( 1 + e^{ -z^l_i } \right)^2 } = a^l_i \left( 1 - a^l_i \right)