(MTR, 5/21/14) We found that if you put a lot of GPUs [specialized graphics processors] together we could make a much bigger neural network—10 billion nodes, with 16 machines instead of 1,000.
We used that same benchmark [images from YouTube videos] that the Google team did. But even though we could train a much larger neural net, we didn’t necessarily get a better cat detector. Right now we can run neural networks that are larger than we know what to do with.
This is a typical situation in Silicon Valley (I described in an earlier post). We are at a point where Machine 1 (exponential growth in computing power) is ahead of Machine 2 (Applications). Most likely, the next S-curve, i.e. a new growth cycle, will begin within the next 5-7 years.
tags: problem, scalability, constraint, machine1, machine2, silicon valley,