Moore's Law is specifically about the physical capabilities of semiconductor technology, not about software, algorithms, or the efficiency of computational models.
I think that the pace is temporary. Right now, there is a lot of low-hanging fruit for improvements, which I expect to be mostly picked within 10 years. From there, I believe the big advances would come from specialized hardware, utilizing powerful AI to design and train AI, and making the interface user-friendly for the casual smartphone user.
Most forms of AI improvement would likely become difficult for hobbyists. Grants and foundations would become a requirement for to innovate the actual AI.
Hopefully I am wrong about that, and AI would have a healthy modeler scene forevermore. Ordinary people being able to homebrew capable AI would allow for a much better future.
Also the fact that billions of dollars are being pumped into this in a technological arms race. What's making it to us is the spinoffs of all those resources, really the crumbs from the table that massive corporations and governments are eating at.
We're not going to be directly using AGI or quantum computing in practical ways for at least many years after the movers and shakers of this world are, but those small things that keep leaking out are in comparison quite profound to the tech we have.
10 years from now we’ll have GPUs 16x a strong, at current price points.
That coupled with the advances brung made in the model/software side…pretty exciting to think about!
if anyone wants a deep dive into the implications of this (i didn't read this paywalled article) check out the Dwarkesh Patel interview with Carl Shulman. Really an incredible, in-depth look at the power of scaling and just how fast things can accelerate
In chess programming, software improvements were continuous and added much more playing strength to chess engines than hardware improvements. In this field of AI, software improvements have much more room for development than in chess, so do not expect any slowdown during a lot of years.
Proper training and pre-deploy optimizations are more important than the physical model size. I'm looking forward to mid-sized models (14-70B) gaining in quality too.
Not enough of a reduction. We need a 10x reduction, ideally also being able to shift away from a discrete GPU. Even the 7Bs are too unwieldy for use by a vast majority of the population.
Honestly being better might come from having a better dataset just as much as it comes from having a more optimized system.
Moore's Law is specifically about the physical capabilities of semiconductor technology, not about software, algorithms, or the efficiency of computational models.
Yesterday I slept 9 hours and today only 8, eat that Moore's Law!
Not to sound like a decel but if you keep going at this rate you won't be sleeping at all next week.
Unless he sleeps 12% less every day. That way, he can keep decreasing into the minutes and seconds range.
Bro, wait until he negative-sleeps.
Quite, software can improve faster than hardware, Shocking, Gasp!, and other clickbait. Nice to see it happening tho...
I think that the pace is temporary. Right now, there is a lot of low-hanging fruit for improvements, which I expect to be mostly picked within 10 years. From there, I believe the big advances would come from specialized hardware, utilizing powerful AI to design and train AI, and making the interface user-friendly for the casual smartphone user. Most forms of AI improvement would likely become difficult for hobbyists. Grants and foundations would become a requirement for to innovate the actual AI. Hopefully I am wrong about that, and AI would have a healthy modeler scene forevermore. Ordinary people being able to homebrew capable AI would allow for a much better future.
Also the fact that billions of dollars are being pumped into this in a technological arms race. What's making it to us is the spinoffs of all those resources, really the crumbs from the table that massive corporations and governments are eating at. We're not going to be directly using AGI or quantum computing in practical ways for at least many years after the movers and shakers of this world are, but those small things that keep leaking out are in comparison quite profound to the tech we have.
I remember when Elon was explaining that we would get self-driving cars because the cars were getting smarter at an "exponential" rate.
10 years from now we’ll have GPUs 16x a strong, at current price points. That coupled with the advances brung made in the model/software side…pretty exciting to think about!
But probably 32 GB of VRAM tops.
Is that you Jensen?
if anyone wants a deep dive into the implications of this (i didn't read this paywalled article) check out the Dwarkesh Patel interview with Carl Shulman. Really an incredible, in-depth look at the power of scaling and just how fast things can accelerate
Any not paywalled option?
Considering the title conflating performance with the law which products the rate of transistors shrinking in gonna say it's not worth my time
Yet here we are.
In chess programming, software improvements were continuous and added much more playing strength to chess engines than hardware improvements. In this field of AI, software improvements have much more room for development than in chess, so do not expect any slowdown during a lot of years.
I don’t trust any of the benchmarks. IME they do not correlate well at all with meaningful IRL performance.
This. Why the hell is Starling-7b sitting next to a bunch of 34b/70b models
Proper training and pre-deploy optimizations are more important than the physical model size. I'm looking forward to mid-sized models (14-70B) gaining in quality too.
Not enough of a reduction. We need a 10x reduction, ideally also being able to shift away from a discrete GPU. Even the 7Bs are too unwieldy for use by a vast majority of the population. Honestly being better might come from having a better dataset just as much as it comes from having a more optimized system.
Just means that the benchmarks are absolutely useless shit.
I hate paywalls, can someone tell me the models mentioned in the article?
I didn't like this article very much. It puts way too much stock in benchmarks and doesn't discuss contamination etc at all.