I asked @ilyasut how to set neural network init. He accidentally replied with a poem: You want to be on the edge of chaos Too small, and the init will be too stable, with vanishing gradients Too large, and you'll be unstable, due to exploding gradients You want to be on the edge

Apr 8, 2019 · 8:16 PM UTC

22
227
20
1,298
Load newest
Replying to @gdb @jackclarkSF @ilyasut
You want The edge Of chaos. Too small, Too stable, Gradients vanishing. Too large, Now unstable, Exploding gradients. You want The edge Of chaos.
12
Replying to @gdb @ilyasut
I'm no poet, but this reminds me of a paper by @suryaganguli "We further show that these initial conditions also lead to faithful propagation of gradients as long as they operate in a special regime known as the edge of chaos." arxiv.org/abs/1312.6120
8
Replying to @gdb @ilyasut
This is my new mantra!!!
Replying to @gdb @ilyasut
Here's a really nice paper working this out for vanilla feed-forward networks. One neat thing they find is that dropout destroys the order/chaos critical point. arxiv.org/pdf/1611.01232.pdf
1
Replying to @gdb @paulg @ilyasut
Good music is always on the edge ... of what though?
Replying to @gdb @suzatweet @ilyasut
This sounds like the kind of output you’d get from an RNN trained with ML literature and self-improvement blog posts :)
2
Replying to @gdb @ilyasut
Riiiiight.
Replying to @gdb @ilyasut
Chasing the gradient_
Replying to @gdb @PolarTash @ilyasut
He's a smart man, Ilya
Replying to @gdb @ilyasut
Isn't it ridiculous that we're answering the problem of intelligence with something we don't even know how to initialize
Load more