I asked @ilyasut how to set neural network init. He accidentally replied with a poem: You want to be on the edge of chaos Too small, and the init will be too stable, with vanishing gradients Too large, and you'll be unstable, due to exploding gradients You want to be on the edge