My new QUAD-GPU folding PC - setup and tuning.

TimTim Southwest PA
edited April 2010 in Folding@Home
Here's my latest project - a PC built purely to fold, with 4 GPUs in it!

It's up and running on 3 GPUs now, I need to do some tuning to get it fully operational.

Parts list:

MSI K9A2 Platinum motherboard
Phenom 9650 quad core cpu, stock 2.3 Ghz speed
2 GB (2 X 1GB) Crucial Ballistix DDR2 PC-8500 1066 Mhz ram, BIOS set to 2.0 volts.
Noctua NH-U9B cpu cooler
80 GB Seagate Barracuda 7200.9 SATA hard drive
XP Home SP3 (YES, it's running IE6 !);D
Coolmax RM-750B aluminum power supply
4 PNY 256 MB 8800 GT graphics cards
The cheapest mid-tower ATX Made in China computer case that Fry's had in stock.

Special software - HFM.net 0.4.8 beta for folding monitoring.

I got the HFM thing figured out, and I started by only having 1 GPU in, then once it was working fine, I added GPU2, then GPU3, etc.

It's stable and folds fine with 3 GPUs in. I'm using the 175.19 drivers, no cpu folding. Cpu usage 75%.

Now there is some tuning and trial and error work to be done.

I can let it go with 3 GPUs in, and it's fine and stable. I leave for work and come back later and it's all good.

But if I put the 4th GPU in the computer, even if it is not doing anything, I can't fold on more than 2 GPUs. If I start up the 3rd folding instance, the computer will shut down in about 30 seconds. I have to turn the power supplies' main switch off and then back on to restart it.

I tried the 196.21 drivers, and they did knock the cpu usage down to almost nothing, but anything more than one folding instance will shut down the computer.

I went back to the 175.19 drivers, and it's fine again on 3 GPUs.

I'm thinking maybe the power supply isn't good enough. It has 3 12 volt rails, 19,18, and 18 amps. I tried to spread the power load as evenly as possible.

Right now the 3 GPUs are folding a P5781 for 2807 PPD, a P5767 for 3210 PPD, and a P10107 for 2343 PPD. Isn't that kind of low? I thought I'd be getting 4000-6000 PPD out of each GPU. I know these drivers are old, but they work, although they might be the problem.

Any ideas for a good stable folding driver version for an 8800 GT 256MB? Or maybe just a weak power supply?

The computer case came with a Raidmax 380 watt unit, is there a way I can hook it up to run 1 or 2 of the GPUs? And the Coolmax handles everything else?

See the gain in my folding on our team list, Tim225 is my folding name. I just got into 34th place on the team. The spike in points in the last 2 days is the new computer doing its work!:rockon:

And HOW do I get my folding signature to display correctly? I thought I had it right but all I can get is this clickable link below.
«1

Comments

  • _k_k P-Town, Texas
    edited March 2010
    You need to do it as an image, Insert Image.

    Sounds like PSU. Max effic. that PSU will do 554 watts on the 12v. That or the board won't take that heavy of traffic over the bus. Kind of a guess.

    So if you swap around the PSU and you can get folding on all 4 running with the older drivers but you still have issues with the newer drivers it is bus traffic crashing the board. I am guessing that with that older chipset that it does not like handling that much traffic that the newer drivers generate because it takes all CPU load and handles it in driver and board traffic. Make sure the board BIOS is updated.
  • TimTim Southwest PA
    edited March 2010
    Thanks _k_, I'll look at what version my current BIOS is.

    Wow, 112 views and only one response so far? Come on people, let's get some ideas on here!
  • Sledgehammer70Sledgehammer70 California
    edited March 2010
    PPD on those cards should be about 3k-3.5k as they are running 112 CUDA cores. Download EVGA Precision and get some OC on those cards and easily break 4k per day.
  • TimTim Southwest PA
    edited March 2010
    I looked it up, on the startup screen it says the BIOS is version 2.8.1030.0032, Fast Trak PDC42819. The actual screen itself says American Megatrends v. 02.61 .

    EDIT - I downloaded a live update utility from MSI, it says my BIOS is version 1.74 .

    EDIT #2 - I used MSI's LiveUpdate tool to download and install the newest BIOS version. Now I have 1.90 instead of 1.74. The notes for the new BIOS only included a couple minor things that it fixed, none of which sounded like the problem I have. But we'll see if it makes any difference.

    The auto update in Windows was pretty easy, it did all the work. I just sat here for 2 minutes thinking how the PC was NOT plugged into one of the battery backup plugs in my UPS.

    But after it did the work and restarted, it all worked fine.

    Hey Sledgehammer, what sort of OC should these 8800 GTs be able to handle? I think they are 600 Mhz stock.
  • Sledgehammer70Sledgehammer70 California
    edited March 2010
    Grab EVGA Precision and Furmark and slowly push them up a few points. Most of those cards can take a 15% increase in clock/shader & memory boost. But what is stable is tough to tell each card may be different and EVGA precision will allow you to adjust each card. even though they are not EVGA cards.
  • LeonardoLeonardo Wake up and smell the glaciers Eagle River, Alaska
    edited March 2010
    Wow, quite a bit here to address:

    1. Nvidia GPU overclocking for the benefit of GPU2 Folding production: Don't mess with video RAM and GPU core clocks. The production gain will be only minimal. The shader clock WILL make a big difference. If I remember correctly, the 8800GT's default shader clock is 1500. You should be abe to get stable shader clocks under Folding between 1675 and 1725 (higher if you have a very good chip), assuming cooling is good and your PSU putting out stable, accurate current.
    a) Use EVGA Precision for GPU clock monitoring and setting. I've used Precision on several brands of Nvidia-based cards with equal success on all. If you do overclock, you must monitor (HFM.NET) the clients to know when you hit the stability wall. Not all work units are created equal. You may think the GPU is rock solid stable, until one day, wham, a new, challenging work unit fails in series.
    b) Do some Internet searches on GPU overclocking. If I am correct, there are break points in the shader clock settings, in that the clock doesn't change until you hit certain ramps: such as 1525 is an indicated shader overclock but an actual clock change does not occur until 1530. (I made up those numbers - I don't remember what the break points are.)

    2. a) Stability problems with four video cards. I've never run four video cards in the same machine - four GPUs, yes, but not four cards - so my my comments here are only partially based on experience. The problems you describe are attributable to the power supply or the motherboard...or both. Many have used your model of motherboard for multiple video card Folding. Not all have been able to get 4 X cards working simultaneously. I would do some Internet searches to find possible solutions. I don't believe anyone on our Folding team has experience with maxing out that board. (Sorry, I wish I could provide you more help than that.)
    b) CPU load should be in low, single digits, with all GPUs Folding. Something is clearly wrong if it goes into double digits for more than just a second, assuming no other programs are running. Most of the ForceWare CUDA drivers have been very good with my multi-GPU Folding boxes. When I'm home this evening, I'll try to remember to post in this thread the drivers that have worked well for me. I'll check when I'm home, but I believe my Nvidia Folders are all running 195.62. Don't use 196.21 - it breaks/disrupts overclocking utilities.
  • TimTim Southwest PA
    edited March 2010
    After getting the 1.90 BIOS update in place, I tried again with the 4 GPUs. Starting up folding one GPU at a time, it's fine with 2 instances going, but when I start up the third one, it crashes out.

    I took out the 4th GPU, and it runs fine on 3.

    It could be a power supply thing, I knew I should have gotten a larger one and a better brand, but they are expensive. I think I'll probably end up putting in a bigger power supply, but I'd like to try everything else first.

    These motherboard designers should build the boards so that if it has 4 GPU slots, it can handle anything that 4 GPUs will throw at it. And then some.

    Leo, let me know what driver version you're running and I'll try that one.
  • Sledgehammer70Sledgehammer70 California
    edited March 2010
    I am running my GTX 295 & a 9800GTX+ I had to do some adjustments to get them all to work perfectly without crashing. In the Nvidia CP make sure multi GPU mode is disabled. Also make sure all 4 cards have a monitor plug to them and are seen on a desktop in your system. Without your system seeing each card & displaying video your system will crash & not work properly.

    In my case my GTX 295 is 2 chips on 2 PCB's so I had to make sure I had a connection on each PCB to get it to work. (Used the HDMI & 1 DVI port) When I used the 2 DVI ports it would lock up my system.

    Each GPU should run at 99% in (EVGA Precision) when properly folding. I know my above issue had the chips running at odd speeds until set it up right. It also helps to run Everest to make sure you have the GPU's numbered... not that it matters to much, but it helps when you launch 1 instance at a time and see the work unit load on the proper GPU in EVGA Precision. Along with assigning proper ID's in the F@H program I also use the GPU tags in the Target location:

    "C:\Program Files (x86)\Folding@home\Folding@home-gpu\Folding@home.exe" -gpu 0
    "C:\Program Files (x86)\Folding@home\Folding@home-gpu\Folding@home.exe" -gpu 1
    "C:\Program Files (x86)\Folding@home\Folding@home-gpu\Folding@home.exe" -gpu 2


    In your case for power issues if you have them... yes you can use your old PSU and jerry-rig it to power on. I know there are plugs you can buy, or guides that show you how to get the PSU to work without other items hooked up.

    As for the 8800GT itself the basic break down on the card is it needs to see 26amps on the 12v rail to power on. Wattage isn't really the issue overall its always the amperage on the 12v rail. While yes Wattage is important overall as these chips start to run at 100%... but to get the cards to power on the amperage needs to be there.
  • LeonardoLeonardo Wake up and smell the glaciers Eagle River, Alaska
    edited March 2010
    195.62 - what I'm running now, no issues.
    196.21 breaks GPU tweaking software.
    196.34 Beta, with fix for problem found in 196.21.
    196.75, recalled by Nvidia after many user reports of video card fan controls no longer working properly.

    What operating system is your rig? If it's Vista or Windows 7, you will need connect each video card to a monitor or install dummy plugs. The plugs are easy to make. If the machine is XP, no plugs or unused monitor connections are needed.
  • Sledgehammer70Sledgehammer70 California
    edited March 2010
    Agreed with the shader clock but increasing the core clock will actually give you a more stable result.
  • TimTim Southwest PA
    edited March 2010
    I'm using XP Home SP3. I've been using the dummy plugs for the extra video cards, 3 75 ohm resistors in DVI-VGA adapters.

    I'm going to try the 195.62 drivers, and the overclocking utility program after that.

    Also, for the second and third video cards, I was using one 4 pin molex connector (the entire wire from the power supply) for each video card. Since they have the 2 4-pin into one 6 pin Y cable adapter, I wonder if it might be better for one branch of the adapter Y cable to go to each rail of the power supply, maybe that will help power distribution a bit. I have to look into using the extra power supply if needed, I just need a way to start it, since the computers' power button can't plug into it.
  • TimTim Southwest PA
    edited March 2010
    I tried the 195.62 drivers, with only 3 GPUs in the slots. I could get 2 of them folding, with nearly 0 cpu usage, but when I went to start up the 3rd instance, the PC would shut down.

    Since it only has 3 GPUs in it, it's not using more power than before, so might it be a bus issue where it can't handle the extra load of the cpu work being done on the graphics card?
  • _k_k P-Town, Texas
    edited March 2010
    PCI-E traffic load would be guess.

    If you want to be all retail about it.
  • Sledgehammer70Sledgehammer70 California
    edited March 2010
    You should also check how many PCI-E lanes each card is seeing. If they drop below 4x lanes per card you might see issues.
  • _k_k P-Town, Texas
    edited March 2010
    When 4 PCI Express x16 slots are all installed, the PCIE x 16 lanes will auto arrange form x16/ x0/ x16/ x0 to x8/ x8/ x8/ x8
  • Sledgehammer70Sledgehammer70 California
    edited March 2010
    Not on every board... On some the 4th slot might drop to 2x
  • _k_k P-Town, Texas
    edited March 2010
    That is straight off the MSI product page.
  • TimTim Southwest PA
    edited March 2010
    I worked on it some more today, and I think it's a bus limitation. Here's why.

    I found out on google that I can jump the green wire to any black wire on the second power supplies' main motherboard connector, and use the switch on the back of the power supply to turn it on and off. I tried installing the 4th GPU with its power coming from the second supply. With the 175.19 drivers, it seemed to work, but when I click on the Folding icon for the 4th GPU, after a few seconds I started seeing horizontal lines in the screen.

    Not really lines, just a distortion in the desktop image, in the shape of lines going across the screen. These lines slowly scrolled upwards, like an old TV set with the vertical hold slightly misadjusted.

    I'm sure at least SOME people here are old enough to remember having to set the vertical hold on a CRT television!

    As soon as I stopped the 4th GPU's folding, the lines in the display stopped.

    I also tried with the 195.62 drivers, and I also tried it with the second power supply powering GPU's #3 and #4. Same thing in each case.

    So it's back on the 175.19 drivers, with 3 GPUs folding, a CPU instance folding, and Task Manager says the CPU is at 100%. The CPU instance in HFM monitor is only reporting 300-350 PPD, so I might just stop it and let the CPU go with a 75% load.

    In a 75 degree room, the CPU temps are 47-50 degrees C. Using the EVGA Precision tuner, the GPU temps were 74-77 degrees C at 38% stock fan speed. I increased the fan speed to 60%, (the fans are still quiet), and knocked 11-14 degrees off each GPU's temperature.

    I'll start experimenting with overclocking the shaders tomorrow.

    I'm disappointed I can't use all 4 GPUs in this PC, is there any way I can adjust the x4 / x8 / x-whatever speeds on the PCI Express slots?

    That dual PSU adapter cable seems like a good idea, instead of buying one really expensive 800-1000 watt PSU, you could buy 2 cheap 400-500 watt units and save money! But it might be more power inefficient in the long run.
  • Sledgehammer70Sledgehammer70 California
    edited March 2010
    No... you can't adjust the bus allocation. You just need a board that supports more lanes.
  • LeonardoLeonardo Wake up and smell the glaciers Eagle River, Alaska
    edited March 2010
    Tim, I admire your tenacity on this. Keep it up, dude!
  • TimTim Southwest PA
    edited March 2010
    Thanks. I picked up another PC for $40 today from someone who bought a new one. I have to look over this one, I think it's a Pentium dual core or something like that, and it has 1 PCI-E slot, so it can put my 4th 8800GT to use. Plus some CPU folding.
  • _k_k P-Town, Texas
    edited March 2010
    Let us know when you have everything up and running plus post of up some ppd screen shots from HFM.NET
  • TimTim Southwest PA
    edited March 2010
    It's been running, see my team folding stats (Tim225) from the last week. Currently in 34th position on the team. And I'm about #6-#7 in daily PPD right now on Team 93.
  • _k_k P-Town, Texas
    edited March 2010
    SCREEN SHOTS I MUST HAVE THEM:bucket:

    Plus if you bother to fill out the fields in HFM it provides a good bit of information on production stats.
  • TimTim Southwest PA
    edited March 2010
    Uh, how do I get a screenshot of HFM 0.4.8?
  • _k_k P-Town, Texas
    edited March 2010
    ctrl+print screen
  • TimTim Southwest PA
    edited March 2010
    Tried that, I don't see where it saved the file on the computer.
  • shwaipshwaip bluffin' with my muffin
    edited March 2010
    alt-printscreen puts a screenshot of the current window on your clipboard. open up paint and hit ctrl-v
  • _k_k P-Town, Texas
    edited March 2010
    Tim.....I would of hoped that you knew to use paint on your own since there have been about 3 threads in the past week where this same question was asked.
  • Sledgehammer70Sledgehammer70 California
    edited March 2010
    I LOL'ed hard on these last posts.
Sign In or Register to comment.