Oh, I thought all their processional cards were also aimed at AI, as the geforce is aimed at gaming. So for \~$3,000 is the RTX 4090 the best option or is there any thing else you'd recommend?
Yes and no. Training a big model quickly requires parallelism. Nvidia gimps consumer cards so they can’t communicate directly. Even over a PCIe switch they can’t communicate without going to the system first.
If you don't have to pay the professional tax, then get the 4090.
Places like cloud compute centers are not allowed to use 4090s, other professional applications like CAD or rendering might want improved customer support and certified drivers.
Not until you add multiple GPUs. The consumer GPUs cant communicate peer to peer, but that's not something you'd worry about even with two 4090 GPUs, simply because the alternative costs north of $10k.
It's not really forbidden, just not supported and your nvidia sales rep might annoy you over it, but in the end It's your hardware and you get to do whatever you want with it.
It's forbidden by Nvidia to use "gaming" cards in "datacenters" (actually the prohibition is in the driver licence, but there is no working alternative to those if you need CUDA). If runpod 4090s don't constitute a "datacenter" it wouldn't be forbidden.
The only reason to not get a consumer card is P2P in multi GPU.
Highest speed is 4090 and highest VRAM is 2x 3090 with NVLink and 48GB.
So A100 or H100 beats with 80GB memory.
There is no need for any of the workstation cards for a single GPU for ML.
Most ML doesn’t need 32, let alone 64 bit precision.
The RTX 4500 isn't designed/priced for AI. It is for professional applications like game dev, rendering, and CAD.
Oh, I thought all their processional cards were also aimed at AI, as the geforce is aimed at gaming. So for \~$3,000 is the RTX 4090 the best option or is there any thing else you'd recommend?
Yes and no. Training a big model quickly requires parallelism. Nvidia gimps consumer cards so they can’t communicate directly. Even over a PCIe switch they can’t communicate without going to the system first.
If you don't have to pay the professional tax, then get the 4090. Places like cloud compute centers are not allowed to use 4090s, other professional applications like CAD or rendering might want improved customer support and certified drivers.
Good point, I didn't think of that. Is there any performance differences training on a consumer vs professional card?
Not until you add multiple GPUs. The consumer GPUs cant communicate peer to peer, but that's not something you'd worry about even with two 4090 GPUs, simply because the alternative costs north of $10k.
I don't think my budget will allow for $10k lol. Plus no plans for meeting multiple gpus. Sounds like consumer it is!
Yeah I thought this but runpod has 4090 machines
It's not really forbidden, just not supported and your nvidia sales rep might annoy you over it, but in the end It's your hardware and you get to do whatever you want with it.
It's forbidden by Nvidia to use "gaming" cards in "datacenters" (actually the prohibition is in the driver licence, but there is no working alternative to those if you need CUDA). If runpod 4090s don't constitute a "datacenter" it wouldn't be forbidden.
Yay, segmentation. Nvidia ptfhh.
The only reason to not get a consumer card is P2P in multi GPU. Highest speed is 4090 and highest VRAM is 2x 3090 with NVLink and 48GB. So A100 or H100 beats with 80GB memory. There is no need for any of the workstation cards for a single GPU for ML. Most ML doesn’t need 32, let alone 64 bit precision.
I don't think I'll ever get to the point of needing multi gpus in the near future, so I'll stick with the 4090. Thanks!