How large neural networks are trained across increasingly massive clusters — cleverly slicing the computation on a wide variety of axes, rematerializing intermediate results, and much more:
OpenAI
 @OpenAI
9 Jun 2022
Techniques for training large neural networks, by @lilianweng and @gdb: openai.com/blog/techniques-f…

Jun 9, 2022 · 4:27 PM UTC

2
4
53
Replying to @gdb
While reading this, it just occurred to me that computation constraints may help us discover how the brain parallelism works, since it's been improving over millions of years to solve it. Eventually, constraints may help us discover brain's "backpropagation" or its alternative.
1
Replying to @gdb
Interesting system of Perception Training. But as I mentioned to another the design is missing a method to analyze Logics/Illogics to dismiss erroneous data and contradictions. This omission is the main reason its easy to abuse AI -- to get it to steal the neighbors kid's toys.
1