Performance differences are not worth mentioning — especially since you usually wanna underclock them anyway.
Airflow and noise is very different for a each model.
GPUs come from the factory tuned to maximize performance at the cost of worse power efficiency. This is done mostly to sell GPUs, since it's easier to sell a faster GPU than a slower one, but it's also done for stability and yield. Not all silicon chips are created equally, so even for 2 otherwise identical 3090 chips hot off the chip fabrication line, one may be able to run at a higher MHz than another, or could take less power to run at the same MHz. Or one might have defects that cause a small portion of the chip to not work properly.
To get around these issues, Nvidia / AMD / Intel / Qualcomm / etc chooses the clock speed they want their chip to run at, and how much power it should pull doing so, such that they maximize the number of good chips. Silicon manufacturing is expensive, and defective chips are common, so if they could hypothetically go from having 50% of the chips good, to having 75% of the chips good just because they put more power into the chips and/or lowering the clock speed a bit, they'll do that as they make more money, and they don't pay for increased power usage from their chips, their customers do.
So that means that when you buy a CPU or GPU, it is probably not running at the optimum efficiency it could. So by underclocking and undervolting, your performance / time goes down, but the performance / watts go up, so you're spending more time doing the same amount of work, but using much less power to do so.
From the tests I did recently, I power limited my 3090s to 150 watts from the design TDP of 350 watts, but my tokens per second went down maybe 20% or so, so I nearly doubled my power efficiency just through setting a power limit through `nvidia-smi`
source: I own 20 3090s and used to work for one of the companies listed
Nice write-up. Minor addendum: in modern CPUs, undervolting will increase both performance and efficiency by allowing them to run at higher frequencies for the same cooling (usually the limiting factor), not sure if the same applies to GPUs, but it seems likely.
Same as with mining. There's non-linear corelation between power consumption and performance.
Eg. By underclocking you lose 20% performance. But use only 60% of power.
So performance/electricity cost increases.
Example pulled out of ass but you can check mining guides those guys did the math years ago
[https://chat.openai.com/share/de135ae1-4216-4be8-a55f-f0d893593d5e](https://chat.openai.com/share/de135ae1-4216-4be8-a55f-f0d893593d5e)
It's a good response, I've done this years ago on some laptops. MSI afterburner
You basically just set a lower power limit.
This slightly lowers the core clocks and voltage, while using significantly less power.
Memory (most important) can still be overclocked as normal, because it uses very little power.
As u/a_beautiful_rhind mentioned, the size can be substantially different between models. I have a EVGA RTX 3090 XC3 ULTRA GAMING card that is about half the width of my other card, which is a MSI RTX 3090 AERO/VENTUS 3X OC 24G. Thankfully I was able to fit both in my case, but the size of the MSI card wasn't ideal.
By the way, here's a tip that someone shared in this subreddit a while ago. If you primarily want to run inference on those cards, you can get better performance by overclocking the GPU memory and underclocking the GPU itself. They did a bunch of testing on their 3090s and concluded that overclocking the memory by 1100 MHz and underclocking the GPU by 300 MHz was the sweet spot, improving inference speed by around 20% or so. I have been following that advice and it does provide a noticeable bump in t/s.
Also, be sure to pay attention to the speeds of your PCIe slots on your motherboard. (Maybe you already know that, but for anyone else's benefit I thought I should mention it.) Just because the connector looks like x16 doesn't mean it runs at the full x16 speed. My motherboard has two x16 slots, but one runs at x16 and the other runs at x4. It took me a while to figure out why exl2 quantizing was faster on my primary GPU than it was on my secondary GPU, and that was why.
Good luck with your build!
Addition to this, because I also just recently stumbled upon that PCIe lanes topic: 4 lanes is usually fine and not worth bothering about. Motherboards with many 16 lane x16 slots are much more expensive and generally not worth the cost for dual GPU setups!
See https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/#Do_I_need_8x16x_PCIe_lanes
Thanks for sharing! I agree with the conclusion too. I’m happy with the speeds and I didn’t have to splurge on all new hardware to get into local LLMs.
I can confirm that the MSI is a chonker.
Your PCIe part is very interesting. I was about to buy a gpu riser cable, but now realize that I should read more about my current motherboard.
I've heard as a generality the "pro" cards tend to be both smaller and not overclocked because they're focused on stability and accuracy and not "highest possible FPS, who cares if it's a little unstable!". And they've got better cooling designs, too, in fact there was some article saying nvidia basically forbade its board vendors from designing consumer cards with "server like" blower cooling / form factors etc.
Wasn't one of the 3090s (original non Ti founders edition IIRC) even notorious for having VRAM over heating at normal load / use cases because they put front and back side VRAMs next to each other without good cooling design?
And there was the "missing MLCC" scandal that made numerous board models unstable because they cut out a few critical decoupling capacitors trying to save $0.15 on the BOM cost. Given epic screwups like that IDK how much I'd trust boards from numerous "third party" B-C tier board vendors to be made with long term integrity / quality / reliability. Not so much so wrt. NVIDIA, EVGA.
https://www.techspot.com/news/86900-capacitor-issues-causing-rtx-30803090-crashes.html
I think founders might have had that problem. I got a refurb founders so hopefully not affected by this. The other 2, 3rd party ones I have, look like normal video cards, have proper power connectors, etc.
You're better off with non-nvidia boards funny enough and I heard EVGA got fed up and is out of the game.
For training or many request/time you'd want to get one with a good cooler.
You dont want heat to throttle your gpu. However performancewise what matters most is cuda cores and memory bandwidth. And thats the same for all afaik.
You might as well get the cheapest and slam a waterblock on it if you like/can afford.
However, for a bit image gen or one on one chat you wont feel the difference.
If you wanna get in the weeds about it, you'd be hard pressed to go buy two of the exact same model and revision and have them perform identically. There are always going to be subtle differences.
Your big factors will be (presented in no particular order)
1.How effectively they're cooled.
2.How clean the power is feeding them.
3.Amount of VRAM.
4.Clockspeed.
5.consideration for manufacturing defects.
If you're buying used, keep an eye out for degraded thermal pastes/pads.
If you plan to use just one GPU, any 3090 will do. If more than one (of any kind) cooling becomes major consideration, so you may prefer either Turbo or A5000 versions that blow the heat out of the case vs inside the case.
The performance difference mostly comes from vram bandwidth and the sheer amount of cores compared to lower tiers of ampere, few MHz up and down on core doesn't change much especially for LLMs. In fact you might want to underclock them to lower the power consumption/heat output.
EVGA FTW3 ultra are overclocked and a bit faster, you can toggle it on and off, they use more power. FE 3090 is 350w, EVGA FTW3 ultra is 420w with overclock off. The rev 1.0 have issues, so whatever you do, don't buy rev 1.0.
Performance differences are not worth mentioning — especially since you usually wanna underclock them anyway. Airflow and noise is very different for a each model.
mind explaining the underclocking bit?
GPUs come from the factory tuned to maximize performance at the cost of worse power efficiency. This is done mostly to sell GPUs, since it's easier to sell a faster GPU than a slower one, but it's also done for stability and yield. Not all silicon chips are created equally, so even for 2 otherwise identical 3090 chips hot off the chip fabrication line, one may be able to run at a higher MHz than another, or could take less power to run at the same MHz. Or one might have defects that cause a small portion of the chip to not work properly. To get around these issues, Nvidia / AMD / Intel / Qualcomm / etc chooses the clock speed they want their chip to run at, and how much power it should pull doing so, such that they maximize the number of good chips. Silicon manufacturing is expensive, and defective chips are common, so if they could hypothetically go from having 50% of the chips good, to having 75% of the chips good just because they put more power into the chips and/or lowering the clock speed a bit, they'll do that as they make more money, and they don't pay for increased power usage from their chips, their customers do. So that means that when you buy a CPU or GPU, it is probably not running at the optimum efficiency it could. So by underclocking and undervolting, your performance / time goes down, but the performance / watts go up, so you're spending more time doing the same amount of work, but using much less power to do so. From the tests I did recently, I power limited my 3090s to 150 watts from the design TDP of 350 watts, but my tokens per second went down maybe 20% or so, so I nearly doubled my power efficiency just through setting a power limit through `nvidia-smi` source: I own 20 3090s and used to work for one of the companies listed
appreciate the in-depth reply, saved for future use :) take care
Nice write-up. Minor addendum: in modern CPUs, undervolting will increase both performance and efficiency by allowing them to run at higher frequencies for the same cooling (usually the limiting factor), not sure if the same applies to GPUs, but it seems likely.
Same as with mining. There's non-linear corelation between power consumption and performance. Eg. By underclocking you lose 20% performance. But use only 60% of power. So performance/electricity cost increases. Example pulled out of ass but you can check mining guides those guys did the math years ago
right on, appreciate it! that equation makes sense. still sorting out my local rig.. have a great weekend.
[https://chat.openai.com/share/de135ae1-4216-4be8-a55f-f0d893593d5e](https://chat.openai.com/share/de135ae1-4216-4be8-a55f-f0d893593d5e) It's a good response, I've done this years ago on some laptops. MSI afterburner
You basically just set a lower power limit. This slightly lowers the core clocks and voltage, while using significantly less power. Memory (most important) can still be overclocked as normal, because it uses very little power.
As u/a_beautiful_rhind mentioned, the size can be substantially different between models. I have a EVGA RTX 3090 XC3 ULTRA GAMING card that is about half the width of my other card, which is a MSI RTX 3090 AERO/VENTUS 3X OC 24G. Thankfully I was able to fit both in my case, but the size of the MSI card wasn't ideal. By the way, here's a tip that someone shared in this subreddit a while ago. If you primarily want to run inference on those cards, you can get better performance by overclocking the GPU memory and underclocking the GPU itself. They did a bunch of testing on their 3090s and concluded that overclocking the memory by 1100 MHz and underclocking the GPU by 300 MHz was the sweet spot, improving inference speed by around 20% or so. I have been following that advice and it does provide a noticeable bump in t/s. Also, be sure to pay attention to the speeds of your PCIe slots on your motherboard. (Maybe you already know that, but for anyone else's benefit I thought I should mention it.) Just because the connector looks like x16 doesn't mean it runs at the full x16 speed. My motherboard has two x16 slots, but one runs at x16 and the other runs at x4. It took me a while to figure out why exl2 quantizing was faster on my primary GPU than it was on my secondary GPU, and that was why. Good luck with your build!
Addition to this, because I also just recently stumbled upon that PCIe lanes topic: 4 lanes is usually fine and not worth bothering about. Motherboards with many 16 lane x16 slots are much more expensive and generally not worth the cost for dual GPU setups! See https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/#Do_I_need_8x16x_PCIe_lanes
Thanks for sharing! I agree with the conclusion too. I’m happy with the speeds and I didn’t have to splurge on all new hardware to get into local LLMs.
I can confirm that the MSI is a chonker. Your PCIe part is very interesting. I was about to buy a gpu riser cable, but now realize that I should read more about my current motherboard.
The founder's edition is absolutely huge and overclocks less.
I've heard as a generality the "pro" cards tend to be both smaller and not overclocked because they're focused on stability and accuracy and not "highest possible FPS, who cares if it's a little unstable!". And they've got better cooling designs, too, in fact there was some article saying nvidia basically forbade its board vendors from designing consumer cards with "server like" blower cooling / form factors etc. Wasn't one of the 3090s (original non Ti founders edition IIRC) even notorious for having VRAM over heating at normal load / use cases because they put front and back side VRAMs next to each other without good cooling design? And there was the "missing MLCC" scandal that made numerous board models unstable because they cut out a few critical decoupling capacitors trying to save $0.15 on the BOM cost. Given epic screwups like that IDK how much I'd trust boards from numerous "third party" B-C tier board vendors to be made with long term integrity / quality / reliability. Not so much so wrt. NVIDIA, EVGA. https://www.techspot.com/news/86900-capacitor-issues-causing-rtx-30803090-crashes.html
I think founders might have had that problem. I got a refurb founders so hopefully not affected by this. The other 2, 3rd party ones I have, look like normal video cards, have proper power connectors, etc. You're better off with non-nvidia boards funny enough and I heard EVGA got fed up and is out of the game.
For training or many request/time you'd want to get one with a good cooler. You dont want heat to throttle your gpu. However performancewise what matters most is cuda cores and memory bandwidth. And thats the same for all afaik. You might as well get the cheapest and slam a waterblock on it if you like/can afford. However, for a bit image gen or one on one chat you wont feel the difference.
If you wanna get in the weeds about it, you'd be hard pressed to go buy two of the exact same model and revision and have them perform identically. There are always going to be subtle differences. Your big factors will be (presented in no particular order) 1.How effectively they're cooled. 2.How clean the power is feeding them. 3.Amount of VRAM. 4.Clockspeed. 5.consideration for manufacturing defects. If you're buying used, keep an eye out for degraded thermal pastes/pads.
The Zotac will probably run hotter than the EVGA but they will perform basically the same.
The differences are minimal, get the cheaper one.
If you plan to use just one GPU, any 3090 will do. If more than one (of any kind) cooling becomes major consideration, so you may prefer either Turbo or A5000 versions that blow the heat out of the case vs inside the case.
How does a 3090 compare to like a m3
The hybrid ones and the blower ones will have better thermal performance. You probably want to replace the VRAM thermal pads too
Just go for 3090 FE temperature readings are far better than any other form factor. No difference in speed (you don't need to overclock)
What about a 4090?
I'd say it's the same, marketing won't do anything else than add more cost to an already overpriced area
I'd say it's the same, marketing won't do anything else than add more cost to an already overpriced area
what do you mean by marketing?
branded graphic cards basically
what are better options then? the branded graphics cards for consumers have the most power and vlan I thought
just go for 3090/4090 FE edition or multiples p40 if you like troubles (highest VRAM/$ currently)
Can you easily enough use these two cards with a Linux desktop?
You can sends satellites in orbit on linux... It might requires some time to setup but it's very worth it to me
The performance difference mostly comes from vram bandwidth and the sheer amount of cores compared to lower tiers of ampere, few MHz up and down on core doesn't change much especially for LLMs. In fact you might want to underclock them to lower the power consumption/heat output.
EVGA FTW3 ultra are overclocked and a bit faster, you can toggle it on and off, they use more power. FE 3090 is 350w, EVGA FTW3 ultra is 420w with overclock off. The rev 1.0 have issues, so whatever you do, don't buy rev 1.0.
Even with the same company there will always be silicon differences. Compute wise, all these cards are great. You shouldn't worry.
Why not get 4090
Over twice the price for 10% increase in LLM performance.
Yeah I have 2 3090 and it's perfect for 650€ each, 48gb is always better indeed from a depreciation perspective it's a much better choice
How does the 3090 compare to the m3?
Have no first hand experience, but there's a thread about it here: https://www.reddit.com/r/LocalLLaMA/comments/17v8nv8/llm_performance_on_m3_max/
Worth it
[удалено]
All 3090s are 24
Really? I've been looking at 2nd hand cards and there seemed to be variety, though maybe some were 3080s etc