A surprising deep learning mystery: Contrary to conventional wisdom, performance of unregularized CNNs, ResNets, and transformers is non-monotonic: improves, then gets worse, then improves again with increasing model size, data size, or training time. openai.com/blog/deep-double-…
94
673
72
2,047
Isn't this the "double descent" phenomenon studied in arxiv.org/abs/1812.11118 and subsequent works?

Reconciling modern machine learning practice and the bias-variance...

Breakthroughs in machine learning are rapidly changing science and society, yet our fundamental understanding of this technology has lagged far behind. Indeed, one of the central tenets of the...

arxiv.org
4
8
126
Replying to @TheGregYang @OpenAI
That's the first citation in the blog post. Note also that the first author, Mikhail Belkin, provided helpful discussions and feedback throughout this work, as mentioned at the bottom.

Dec 5, 2019 · 5:42 PM UTC

1
1
34
Replying to @gdb @TheGregYang @OpenAI
Hiding the citation in a hyperlink is the wrong way to do this. Readers might assume OpenAI were the first to discover this. Text of blog post should assign credit liberally/explicitly, and explain that the (valuable!) contribution is showing this empirically for CNNs etc.
1
3
62