G34 Folding Rig w/ water cooling parts list

Tushon

Looking for some guidance/thumbs up/down on my plan for this rig. I have to either sell off some existing parts or hold off on this build till late 2012/early 2013, but thinking/talking about it now should suffice for the moment. Water cooling because this will be in an apartment and I want the noise to drop from my existing setup. Here is my current plan:

Need to purchase
4x Opteron 6174 (used ~$500 each)
1x Xigmatech Elysium $190-220 (reputable)
1x Supermicro H8QGi+-F ~$655 (~100 cheaper than the H8QGi-F for only losing some PCI-E slots that I wouldn't be using anyways)
1x 16GB kit Corsair Vengeance 4x4GB DDR3-1600 CL7 RAM (net some significant improvements in PPD vs CL8 or CL9 or DDR3-1333 CL7)

Possibly (probably?) a fan controller

Owned
Solid 950W PSU currently in my main rig, I also have an 750 or 850 spare that will go into gaming rig when 950W moves to folding rig

Water cooling (EDIT prices went up lulllllz ... go big or go home, I guess)
4x Koolance CPU-370 $104 each, with pair of these compression fittings and the G34 adapter bolts listed
1x FrozenQ Liquid Fusion Dual Bay Reservoir w/ Swiftech MCP655 Series Pump Installed $260 (may end up being out of my budget, but a boy can dream for such a cool solution
1x Black Ice SR1 Low Air Flow $150

Various fittings for attaching hoses to blocks/res/pump and all that has to match. details can be worked out later, but I'm thinking 1/2" OD/3/8" ID or a size down (whatever matches fittings) with compression fittings. I don't have any reason to get the quick release stuff as long as I set up system to be drained easily.

General comments: I may change out fans in the case for quieter ones, but I need to look into it more. I think there would only be 1-2 fans anyways (side panel and rear) due to using 1-2 rads in the case. If I only used a single rad, I'm thinking I'd need to have 2 fans in front, running slowly, but will definitely like advice in this regard.


Questions:
Is RAM total is overkill? Could I drop to 4x2GB and be okay for bigadv folding? --EDIT: looks like I should stay at 16GB

Is the single 420 res enough for cooling w/ moderate overclocking via the tear custom BIOS? Most of the threads I looked at on various sites had a 360 in the front and 360 in the top, but I'm assuming those people are also cooling GPU(s). That kills the HDD space which I would need to replace my current folding/storage server/FTP box. In the same vein, I'm not entirely sure what the differences are between the three rads I listed besides the first one using 140mm fans (and the case is supposed to have room for a rad and 140mm fans in the top alone).

Does 950W sound like enough for the purposes stated? Assume no GPU folding, but if I have enough room for one, I could play with GPU3 via WINE and that would be nice.

I have almost no knowledge of water cooling outside of what I've read here and elsewhere, so please say "that is silly" or "that works". Resevoir vs t line is something I don't know pro/con for.

Paging @Leonardo, @_k_, @csimon for starters

csimon

Did you mean 1/2" or 3/8" ID on that tubing?
I'm sure the Maze4 is an adequate block but I shy away from acrylic since they crack easiest when fastened too tightly ...but if you do go that route be very careful. Also, I thought that 12GB was the minimum for -bigadv but I have no documentation to support that at the moment.

I'll look at this closer later when it isn't so late. Good luck ...and nice rig btw.

Tushon

figured 1/2" OD, so 3/8 ID, right? Something like this (just a first example)

The other G34 block I saw on first pass was this one.

I think I'll end up building a rig that is UV reactive, but it'll need an a master "lights out" switch.

Straight_Man

@csimon 12 processor cores (per folding) going on 16 for -bigadv WUs. I personally would, with 4 Opteron 6174s, provide 4GB RAM per CPU chip or 1 GB per core. For heavy server use, folks are suggesting 8GB per CPU for 6174s or more recent CPUs.

Wish I could say that I had the funds or cooling for this, but I do not have that luxury.

Tushon

@csimon 12 processor cores (per folding) going on 16 for -bigadv WUs. I personally would, with 4 Opteron 6174s, provide 4GB RAM per CPU chip or 1 GB per core. For heavy server use, folks are suggesting 8GB per CPU for 6174s or more recent CPUs.

Wish I could say that I had the funds or cooling for this, but I do not have that luxury.

Definitely a long term goal for me, and I'll be selling multiple components to get there as it will be replacing my fileserver rig and my no-longer-bigadv rig.

I'll leave this here for comparison thoughts (he is building that loop to cool the same load)

I also found my old frozen-cpu wishlist that I had put together and have updated my list up there accordingly. A lot of the parts I'm looking at are using 1/4 threaded parts. I presume that means 1/4" opening, so is there a benefit to using 1/2" OD tubing with that? Maybe something smaller would be better. I guess you could go bigger (and therefore slower therefore quieter?) with a different top like this

csimon

This isn't true in every case but 1/2"OD tubing with 3/8"ID ...think of this as the female part.
3/8"OD barb fitting with 1/4"ID ...think of this as the male.

This should mate just fine if this is the case ...so double check sizes.

Also, copper/aluminum>delrin>plexi.

Straight_Man

Old hydraulic flow/plumbing trick-- when diameter decreases, pressure increases. So, flow at your elbows and possibly through your plate's flow area will at least equal flow through tube. Plumbers use this to maintain adequate pressure piping by designing in decreases in diameter to increase pressure. Water mains go to smaller main feed pipe into house, and the mains can be low pressure flow while pressure into house is enough to flow through house and out faucets, etc.

Basically, the cross-section area of flow tube is 1.5 times flow through connector where they meet , so pressure will be at least 1.5 times more in connector versus in tube.

_k

Why are you going with water instead of air which is going to cut out at least 600$ from your cost?

G 1/4 BSP does not mean 1/4 ID, it is the threading used you can still buy whatever fittings you want. The G1/4 thread is what is used mostly for liquid cooling parts now.

If you want to go with the H8QGi+-F I would jump onto [H] and find out if their hacked overclocking BIOS will work on that board otherwise I would pick up the same board that Leo is running.

More RAM.

PSU will be enough.

Tushon

This isn't true in every case but 1/2"OD tubing with 3/8"ID ...think of this as the female part.
3/8"OD barb fitting with 1/4"ID ...think of this as the male.

This should mate just fine if this is the case ...so double check sizes.

Also, copper/aluminum>delrin>plexi.

Roger that. I thought that was how it worked, but was mainly asking: is there a reason to use 1/2" vs 3/8" vs 1/4" OD. I was thinking 1/2" for higher volume with a lower speed, so lower noise generated by pump. The rule with metal is to make sure it is the same throughout, correct? As in, choose all fittings to be the same metal. (@Straight_Man thanks for the example)

Why are you going with water instead of air which is going to cut out at least 600$ from your cost?

The price of silence is great. I may go back and do the watercooling later, throw some Hyper 212+ in there initially.

G 1/4 BSP does not mean 1/4 ID, it is the threading used you can still buy whatever fittings you want. The G1/4 thread is what is used mostly for liquid cooling parts now.

Roger. /derp

If you want to go with the H8QGi+-F I would jump onto [H] and find out if their hacked overclocking BIOS will work on that board otherwise I would pick up the same board that Leo is running.

According this post, it is good to go ("any flavor of that model number"). I picked it because it was $100 less due to the inclusion of additonal PCI-E slots on the H8QGi-F (non +). The bonus round was that you can leave BMC (a remote monitoring tool) on for the H8QGi* model and cannot on the H8QGL (earlier in the thread)

More RAM.

16GB isn't enough for bigadv + file serving + ftp?

PSU will be enough.

Good deal.

_k

16GB is enough for bigadv. I think the most I ever saw my machine eat was 17% of 16GB, generally I try and go with 1GB+ per core because it is cheap and then you can one shot that simple issue. I have changed RAM on my main 3 times.

Water doesn't mean silent. Given you might be running your main rig in the same room as the server I would spend time looking at how loud that is and how loud the fans are on the PSU, the case, and the CM HS fans are. Buying one of the fans used on the sink is a cheap investment to do research, plus great reason to buy a dba meter.

Buy all Cu parts. With that many CPUs and a possible OC I would be interested in a 120.4, 140.4, or two rads. All depends on how serious you are about temperatures and noise. Trying to make your water as quiet as possible means low volume and low pressure fans plus making sure ALL air is out of your loop and it is staying clear.

I almost bought that case. I have a Switch 810 that should be everything you want and is cheaper, you can come check it out. The switch actually has 120mm HSFs in it.

Tushon

Looking at the pics with an ATX mobo in Switch 810, I don't think the bottom of SWTX mobo would clear PSU. They use 3 rads in their pics, not sure if 140.3 or 120.3, but it is x.3, x.2 on the bottom (so it would have to be elevated or other raised off carpet which would probably be good anyways) and x.1 on the rear vent.

I am looking at WC'ing main rig as well (and prior to the build and watercooling of the folding rig).

The biggest chunk of change up there is the change from $46 water blocks to $104. If I can find a good one that is known to cover the whole CPU of G34 socket, I'll go down.

_k

Top is 140.3 or 120.3 Bottom is the same but .2 and you loose the bottom HDD cage.

shwaip

Because you're building a server, not a machine you need to access, have you considered going air and putting it in a closet somewhere? That way you get quiet + cheap.

(I also understand the desire to play with water cooling, just throwing it out there for ya )

Tushon

Oh I do appreciate it. 4 Hyper 212+ would be hella cheaper (even 4 Noctua's would be over 50% less). I may have to satiate my desire for WC with my main rig and go air for this build but it is way down the line so I won't be making that decision for a while. If it was a normal file server, I wouldn't have any qualms about sticking in a closet. However, since this will be 4 procs outputting a ton of heat, wouldn't cool air flow be an issue in a closet? The only two I would have available currently are a coat closet and walk in in bedroom

@_k_, you'd have to add an inch towards the front and 3 down on that mobo to hit the SWTX size. I don't see that case working for this use.

Leonardo

My comments pertain to workstation usage - Folding@Home. I have no experiencing configuring a server for traditional server roles. I realize I'm reiterating what some of you guys already posted.

I just finished a 4X6174/H8QGi+-F build:
* if you enjoy the handiwork and engineering of installing a water cooling system, hey, go for it; the 4 X Hyper 212+ configuration works great; it's a super performer in both cooling efficiency and low noise; you'll get more noise from the little buzzing chipset fan than from the four Cooler Master 120s; they may be inexpensive, but the Hyper 212+ is impressive and the fan blade designs really work - low noise, high air flow; I have 212s on two G34 builds now - I can't say enough good things about them; the Cooler Master store has "refurbished" 212+s on sale every now and then for as little as about $15. We have not been able to determine any difference between the refurbs and the full retail units other than packaging flaws.
* yes, the [H] "Next Generation" BIOS (SMNGOC) works with all Supermicro G34 'i'/'6' and 'L' series 4XG34 boards.
* populate four channels (four DIMMs) for each G34 socket; quantity of RAM modules matters, quantity, as in 32GB vs. 16GB makes no difference at all
* another enclosure option for an SWTX build is the Cooler Master HAF 932; you just need to drill some additional standoff holes, bend one of the upper PSU supports out the way, and move the top fan to the outside-top of the case; I'm running my new system in a 932; I highly recommend it...except for its butt ugly looks - I swapped out the 932's side cover for the cover on my HAF-X - ahh, so much better
* here's the configuration of my 6174-based system, if it's of use to you:
Motherboard: Supermicro H8QGi+-F, [H] SMNGOC BIOS
CPU: 4 X AMD Opteron, Magny Cours, 6174, 2.2GHz @ 2.42GHz
PSU: Enermax Revolution 950W 85+
Memory: G.Skill, Ripjaws, DDR3 1333, CL 7, 32GB
Video Card/GPU: Matrox 200, integrated
CPU Heatsink: 4 X Cooler Master Hyper 212+, custom mounting hardware, a la the "Musky mod"
Cooler Master HAF 932, modified
.......last, but not least, LED and neon bling

Leonardo

temps:
CPU core temps--
6174 system, 2.2@2.42GHz: 33-44C, ambient 70-74F, full load, overclocked, Folding
6136 system (4 X 8-core), 2.3@2.7GHz: 32-46C, ambient 72-78F, full load, overclocked, Folding

Both systems run on native Linux, Ubuntu 10.10 desktop. Access for both is mainly remote, through Teamviewer or Windows 7 networking.

You might want to consider Magny Cours 6166HE. They overclock very well, allow voltage adjustments, and are typically $100 less expensive than 6174s.

Tushon, I didn't want to dominate your thread, just to give some real world data and provide ideas for any prospective G34 4X builders. Don't hesitate to reach out. I'll help where I can. Team 93, baby!

Tushon

No worries about dominating the thread. That kind of info is exactly what I was looking for! With that kind of review about the Hyper 212 in this exact use case, I'm thinking that water cooling should be out for now. If I decide it is too noisy after WC'ing my main rig (assuming I get around to that, but I probably will since the GPU folding is the noisiest part) that it is still too noisy, I don't have any problem dropping from a $20 after rebate to $10 plus shipping for used or something like that. I think @_k_ and I spoke about getting a custom Tech Tray but I don't know if he went through with it. I'd be curious how loud the machine would be in open air vs a case like the Elysium or HAF 932.

@Leonardo, do you have any numbers you can point at for folding PPD of 6166HE vs 6174? Is there any "real" difference?

RE workstation vs server: I don't see any real differences between the two in terms of performance and GUI will be better for controlling some features like FTP that were a pain to (attempt to) configure via command line. I really should say "server" because the only server roles it performs is file storage on my network and some FTP out to friends.

Tushon

Worth a separate comment: insanely detailed guide with product rec's on Overclock.net

Leonardo

Damn, lot's of questions. (but I like it!!!)
* you'll have to look at 4P enthusiasts threads for specific 6166 production numbers. At stock, no, they won't keep up with 6174s, but they overclock relatively very well
* open air is louder than inside a HAF 932; the CM fans are not bad at all - sort of a rushing sound, but that stinky 40mm chipset fan is buzzy, high pitched, arrggghh (I will be attempting a board mod this weekend just for that, we'll see); side fan on the 932 - i removed it to see what, if any core temp differences there would be. It was about the same, but I found with the fan in place, spinning, it muffled the the sound emanating from the chipset and CPU fans
* I don't know at what threshold - OC frequency - one would hit the effective cooling limits of the Hyper 212+ and need to move to water; my system won't clock that high; I think it's a motherboard vDroop or CPU core undervolting issue, not cooling; nonetheless, the 10% overclock is still better than factory default, so I'm still quite happy; I do know that most of guys getting really high reference clock frequencies on the Supermicro 4X boards are just using air
* if you are interested in bolt-on, made-for-G34 heatsink/fans, I've got four screamers I could sell you cheap

_k

@Tushon Did you get your gear yet?

Tushon

Yeah, it arrived last night. I'm cleaning and taking build log pics tonight

Tushon

So I ended up getting this, on @_k_'s recommendation and with @Leonardo's vouch (I think). I'll be holding off on watercooling for a long while (more likely cooling my main rig instead) and putting together this beast as soon as possible. Still working on getting all the stuff for custom G34 mounting solution for the Hyper 212+ that I'm using since apparently only one store in DFW sells anything like the KV Steel Shelf Standard I need. I'm going to look tomorrow and make sure they have the right item and hopefully I'll have build completed tomorrow night.

Leonardo

Looking forward to the presence of that beast in the machine room of Team 93!