My new QUAD-GPU folding PC - setup and tuning.
Tim
Southwest PA Icrontian
Here's my latest project - a PC built purely to fold, with 4 GPUs in it!
It's up and running on 3 GPUs now, I need to do some tuning to get it fully operational.
Parts list:
MSI K9A2 Platinum motherboard
Phenom 9650 quad core cpu, stock 2.3 Ghz speed
2 GB (2 X 1GB) Crucial Ballistix DDR2 PC-8500 1066 Mhz ram, BIOS set to 2.0 volts.
Noctua NH-U9B cpu cooler
80 GB Seagate Barracuda 7200.9 SATA hard drive
XP Home SP3 (YES, it's running IE6 !);D
Coolmax RM-750B aluminum power supply
4 PNY 256 MB 8800 GT graphics cards
The cheapest mid-tower ATX Made in China computer case that Fry's had in stock.
Special software - HFM.net 0.4.8 beta for folding monitoring.
I got the HFM thing figured out, and I started by only having 1 GPU in, then once it was working fine, I added GPU2, then GPU3, etc.
It's stable and folds fine with 3 GPUs in. I'm using the 175.19 drivers, no cpu folding. Cpu usage 75%.
Now there is some tuning and trial and error work to be done.
I can let it go with 3 GPUs in, and it's fine and stable. I leave for work and come back later and it's all good.
But if I put the 4th GPU in the computer, even if it is not doing anything, I can't fold on more than 2 GPUs. If I start up the 3rd folding instance, the computer will shut down in about 30 seconds. I have to turn the power supplies' main switch off and then back on to restart it.
I tried the 196.21 drivers, and they did knock the cpu usage down to almost nothing, but anything more than one folding instance will shut down the computer.
I went back to the 175.19 drivers, and it's fine again on 3 GPUs.
I'm thinking maybe the power supply isn't good enough. It has 3 12 volt rails, 19,18, and 18 amps. I tried to spread the power load as evenly as possible.
Right now the 3 GPUs are folding a P5781 for 2807 PPD, a P5767 for 3210 PPD, and a P10107 for 2343 PPD. Isn't that kind of low? I thought I'd be getting 4000-6000 PPD out of each GPU. I know these drivers are old, but they work, although they might be the problem.
Any ideas for a good stable folding driver version for an 8800 GT 256MB? Or maybe just a weak power supply?
The computer case came with a Raidmax 380 watt unit, is there a way I can hook it up to run 1 or 2 of the GPUs? And the Coolmax handles everything else?
See the gain in my folding on our team list, Tim225 is my folding name. I just got into 34th place on the team. The spike in points in the last 2 days is the new computer doing its work!
And HOW do I get my folding signature to display correctly? I thought I had it right but all I can get is this clickable link below.
It's up and running on 3 GPUs now, I need to do some tuning to get it fully operational.
Parts list:
MSI K9A2 Platinum motherboard
Phenom 9650 quad core cpu, stock 2.3 Ghz speed
2 GB (2 X 1GB) Crucial Ballistix DDR2 PC-8500 1066 Mhz ram, BIOS set to 2.0 volts.
Noctua NH-U9B cpu cooler
80 GB Seagate Barracuda 7200.9 SATA hard drive
XP Home SP3 (YES, it's running IE6 !);D
Coolmax RM-750B aluminum power supply
4 PNY 256 MB 8800 GT graphics cards
The cheapest mid-tower ATX Made in China computer case that Fry's had in stock.
Special software - HFM.net 0.4.8 beta for folding monitoring.
I got the HFM thing figured out, and I started by only having 1 GPU in, then once it was working fine, I added GPU2, then GPU3, etc.
It's stable and folds fine with 3 GPUs in. I'm using the 175.19 drivers, no cpu folding. Cpu usage 75%.
Now there is some tuning and trial and error work to be done.
I can let it go with 3 GPUs in, and it's fine and stable. I leave for work and come back later and it's all good.
But if I put the 4th GPU in the computer, even if it is not doing anything, I can't fold on more than 2 GPUs. If I start up the 3rd folding instance, the computer will shut down in about 30 seconds. I have to turn the power supplies' main switch off and then back on to restart it.
I tried the 196.21 drivers, and they did knock the cpu usage down to almost nothing, but anything more than one folding instance will shut down the computer.
I went back to the 175.19 drivers, and it's fine again on 3 GPUs.
I'm thinking maybe the power supply isn't good enough. It has 3 12 volt rails, 19,18, and 18 amps. I tried to spread the power load as evenly as possible.
Right now the 3 GPUs are folding a P5781 for 2807 PPD, a P5767 for 3210 PPD, and a P10107 for 2343 PPD. Isn't that kind of low? I thought I'd be getting 4000-6000 PPD out of each GPU. I know these drivers are old, but they work, although they might be the problem.
Any ideas for a good stable folding driver version for an 8800 GT 256MB? Or maybe just a weak power supply?
The computer case came with a Raidmax 380 watt unit, is there a way I can hook it up to run 1 or 2 of the GPUs? And the Coolmax handles everything else?
See the gain in my folding on our team list, Tim225 is my folding name. I just got into 34th place on the team. The spike in points in the last 2 days is the new computer doing its work!
And HOW do I get my folding signature to display correctly? I thought I had it right but all I can get is this clickable link below.
0
Comments
Sounds like PSU. Max effic. that PSU will do 554 watts on the 12v. That or the board won't take that heavy of traffic over the bus. Kind of a guess.
So if you swap around the PSU and you can get folding on all 4 running with the older drivers but you still have issues with the newer drivers it is bus traffic crashing the board. I am guessing that with that older chipset that it does not like handling that much traffic that the newer drivers generate because it takes all CPU load and handles it in driver and board traffic. Make sure the board BIOS is updated.
Wow, 112 views and only one response so far? Come on people, let's get some ideas on here!
EDIT - I downloaded a live update utility from MSI, it says my BIOS is version 1.74 .
EDIT #2 - I used MSI's LiveUpdate tool to download and install the newest BIOS version. Now I have 1.90 instead of 1.74. The notes for the new BIOS only included a couple minor things that it fixed, none of which sounded like the problem I have. But we'll see if it makes any difference.
The auto update in Windows was pretty easy, it did all the work. I just sat here for 2 minutes thinking how the PC was NOT plugged into one of the battery backup plugs in my UPS.
But after it did the work and restarted, it all worked fine.
Hey Sledgehammer, what sort of OC should these 8800 GTs be able to handle? I think they are 600 Mhz stock.
1. Nvidia GPU overclocking for the benefit of GPU2 Folding production: Don't mess with video RAM and GPU core clocks. The production gain will be only minimal. The shader clock WILL make a big difference. If I remember correctly, the 8800GT's default shader clock is 1500. You should be abe to get stable shader clocks under Folding between 1675 and 1725 (higher if you have a very good chip), assuming cooling is good and your PSU putting out stable, accurate current.
a) Use EVGA Precision for GPU clock monitoring and setting. I've used Precision on several brands of Nvidia-based cards with equal success on all. If you do overclock, you must monitor (HFM.NET) the clients to know when you hit the stability wall. Not all work units are created equal. You may think the GPU is rock solid stable, until one day, wham, a new, challenging work unit fails in series.
b) Do some Internet searches on GPU overclocking. If I am correct, there are break points in the shader clock settings, in that the clock doesn't change until you hit certain ramps: such as 1525 is an indicated shader overclock but an actual clock change does not occur until 1530. (I made up those numbers - I don't remember what the break points are.)
2. a) Stability problems with four video cards. I've never run four video cards in the same machine - four GPUs, yes, but not four cards - so my my comments here are only partially based on experience. The problems you describe are attributable to the power supply or the motherboard...or both. Many have used your model of motherboard for multiple video card Folding. Not all have been able to get 4 X cards working simultaneously. I would do some Internet searches to find possible solutions. I don't believe anyone on our Folding team has experience with maxing out that board. (Sorry, I wish I could provide you more help than that.)
b) CPU load should be in low, single digits, with all GPUs Folding. Something is clearly wrong if it goes into double digits for more than just a second, assuming no other programs are running. Most of the ForceWare CUDA drivers have been very good with my multi-GPU Folding boxes. When I'm home this evening, I'll try to remember to post in this thread the drivers that have worked well for me. I'll check when I'm home, but I believe my Nvidia Folders are all running 195.62. Don't use 196.21 - it breaks/disrupts overclocking utilities.
I took out the 4th GPU, and it runs fine on 3.
It could be a power supply thing, I knew I should have gotten a larger one and a better brand, but they are expensive. I think I'll probably end up putting in a bigger power supply, but I'd like to try everything else first.
These motherboard designers should build the boards so that if it has 4 GPU slots, it can handle anything that 4 GPUs will throw at it. And then some.
Leo, let me know what driver version you're running and I'll try that one.
In my case my GTX 295 is 2 chips on 2 PCB's so I had to make sure I had a connection on each PCB to get it to work. (Used the HDMI & 1 DVI port) When I used the 2 DVI ports it would lock up my system.
Each GPU should run at 99% in (EVGA Precision) when properly folding. I know my above issue had the chips running at odd speeds until set it up right. It also helps to run Everest to make sure you have the GPU's numbered... not that it matters to much, but it helps when you launch 1 instance at a time and see the work unit load on the proper GPU in EVGA Precision. Along with assigning proper ID's in the F@H program I also use the GPU tags in the Target location:
"C:\Program Files (x86)\Folding@home\Folding@home-gpu\Folding@home.exe" -gpu 0
"C:\Program Files (x86)\Folding@home\Folding@home-gpu\Folding@home.exe" -gpu 1
"C:\Program Files (x86)\Folding@home\Folding@home-gpu\Folding@home.exe" -gpu 2
In your case for power issues if you have them... yes you can use your old PSU and jerry-rig it to power on. I know there are plugs you can buy, or guides that show you how to get the PSU to work without other items hooked up.
As for the 8800GT itself the basic break down on the card is it needs to see 26amps on the 12v rail to power on. Wattage isn't really the issue overall its always the amperage on the 12v rail. While yes Wattage is important overall as these chips start to run at 100%... but to get the cards to power on the amperage needs to be there.
196.21 breaks GPU tweaking software.
196.34 Beta, with fix for problem found in 196.21.
196.75, recalled by Nvidia after many user reports of video card fan controls no longer working properly.
What operating system is your rig? If it's Vista or Windows 7, you will need connect each video card to a monitor or install dummy plugs. The plugs are easy to make. If the machine is XP, no plugs or unused monitor connections are needed.
I'm going to try the 195.62 drivers, and the overclocking utility program after that.
Also, for the second and third video cards, I was using one 4 pin molex connector (the entire wire from the power supply) for each video card. Since they have the 2 4-pin into one 6 pin Y cable adapter, I wonder if it might be better for one branch of the adapter Y cable to go to each rail of the power supply, maybe that will help power distribution a bit. I have to look into using the extra power supply if needed, I just need a way to start it, since the computers' power button can't plug into it.
Since it only has 3 GPUs in it, it's not using more power than before, so might it be a bus issue where it can't handle the extra load of the cpu work being done on the graphics card?
If you want to be all retail about it.
I found out on google that I can jump the green wire to any black wire on the second power supplies' main motherboard connector, and use the switch on the back of the power supply to turn it on and off. I tried installing the 4th GPU with its power coming from the second supply. With the 175.19 drivers, it seemed to work, but when I click on the Folding icon for the 4th GPU, after a few seconds I started seeing horizontal lines in the screen.
Not really lines, just a distortion in the desktop image, in the shape of lines going across the screen. These lines slowly scrolled upwards, like an old TV set with the vertical hold slightly misadjusted.
I'm sure at least SOME people here are old enough to remember having to set the vertical hold on a CRT television!
As soon as I stopped the 4th GPU's folding, the lines in the display stopped.
I also tried with the 195.62 drivers, and I also tried it with the second power supply powering GPU's #3 and #4. Same thing in each case.
So it's back on the 175.19 drivers, with 3 GPUs folding, a CPU instance folding, and Task Manager says the CPU is at 100%. The CPU instance in HFM monitor is only reporting 300-350 PPD, so I might just stop it and let the CPU go with a 75% load.
In a 75 degree room, the CPU temps are 47-50 degrees C. Using the EVGA Precision tuner, the GPU temps were 74-77 degrees C at 38% stock fan speed. I increased the fan speed to 60%, (the fans are still quiet), and knocked 11-14 degrees off each GPU's temperature.
I'll start experimenting with overclocking the shaders tomorrow.
I'm disappointed I can't use all 4 GPUs in this PC, is there any way I can adjust the x4 / x8 / x-whatever speeds on the PCI Express slots?
That dual PSU adapter cable seems like a good idea, instead of buying one really expensive 800-1000 watt PSU, you could buy 2 cheap 400-500 watt units and save money! But it might be more power inefficient in the long run.
Plus if you bother to fill out the fields in HFM it provides a good bit of information on production stats.