Derivative

Loss function#

Mean Square Error#

Loss=12โˆ‘i(yiโˆ’aiL)2Loss = \dfrac{ 1 }{ 2 } \sum_i \left( y_i - a^L_i \right)^2
โˆ‚Lossโˆ‚aiL=โˆ’(yiโˆ’aiL)\dfrac{ \partial Loss }{ \partial a^L_i } = - \left( y_i - a^L_i \right)

Binary Cross-Entropy#

Loss=โˆ‘i[โˆ’yilogโกaiLโˆ’(1โˆ’yi)logโก(1โˆ’aiL)]Loss = \sum_i \left[ -y_i \log a^L_i - (1 - y_i) \log \left( 1 - a^L_i \right) \right]
โˆ‚Lossโˆ‚aiL=โˆ’(yiโˆ’aiL)aiL(1โˆ’aiL)\dfrac{\partial Loss}{\partial a^L_i} = \dfrac{ - \left( y_i - a^L_i \right)}{a^L_i \left( 1 - a^L_i \right) }

Activation#

logistic#

ail=ฯƒ(zil)=logistic(zil)=11+eโˆ’zila^l_i = \sigma ( z^l_i ) = logistic ( z^l_i ) = \dfrac{ 1 }{ 1 + e^{ -z^l_i } }
โˆ‚ฯƒ(zil)โˆ‚zil=eโˆ’zil(1+eโˆ’zil)2=ail(1โˆ’ail)\dfrac{\partial \sigma ( z^l_i )}{\partial z^l_i } = \dfrac{ e^{ -z^l_i } }{ \left( 1 + e^{ -z^l_i } \right)^2 } = a^l_i \left( 1 - a^l_i \right)
Last updated on