From Richard Sutton (incompleteideas.net/), an essay on the repeated historical finding that computational scale has always beaten cleverness in AI (and some commentary on why this is such a hard-to-accept fact): incompleteideas.net/IncIdeas…

Mar 14, 2019 · 11:02 PM UTC

12
82
8
274
Load newest
Replying to @gdb
@gdb does this worry you? Seems like this is the case of "rich get richer" - largest incumbents have the most compute & data, can do AI better than everyone else. How do ML startups build a moat in these circumstances?
1
Replying to @gdb
Rich Sutton also elaborated on it at Montreal 2017 Summer School: medium.com/@ranko.mosic/prof…

TD Learning — Richard Sutton Presentation at Reinforcement Learning Summer School, Montreal 2017

These are highlights and slides from Prof. Sutton’s lecture:

ranko-mosic.medium.com
This tweet is unavailable
Agreed!
Greg Brockman
 @gdb
15 Mar 2019
Replying to @etzioni
Both are important! Look at GPT-2 for instance — that's a general-purpose architectural improvement (i.e. the Transformer) run at massive scale. One interesting point from the essay is that scale gets a bad rap — doing the reverse isn't a good way of fixing the problem!
1
1
more replies