Just day before yesterday I was working on the backprop assignment of CS231N. The question asked to increase the accuracy on CIFAR10 dataset by cross validation. I added another hidden layer (totalling to 3) and the gradients just seemed to reach in the order of 1e-07 in relu activation. After tinkering a lot with the network, I changed the standard deviation of the weight initialization from 1e-4 to 0.1. This seemed to fix things nicely.

I believe the most rewarding part on learning how backprop works is the fact that you learn to appreciate it’s beauty.