From Richard Sutton (incompleteideas.net/), an essay on the repeated historical finding that computational scale has always beaten cleverness in AI (and some commentary on why this is such a hard-to-accept fact): incompleteideas.net/IncIdeas…
12
82
8
274
Agreed!
Both are important! Look at GPT-2 for instance — that's a general-purpose architectural improvement (i.e. the Transformer) run at massive scale.
One interesting point from the essay is that scale gets a bad rap — doing the reverse isn't a good way of fixing the problem!
Mar 15, 2019 · 3:52 AM UTC
1
1
