Cheap 16core Xeon Node - Pros/Cons?

[jensandersen]

Feeding the Noctuas enough cold air and let it seep out where possible, seems to work better than trying to remove the hot and not having enough cold air comming in.

Any recommendations on one good 120mm that could pull in lots of cold air but without sounding like a hair dryer?

 

Yeah that too Scott, I'm not sure if the one CPU running a bit hotter is the one I added the tiny bit extra to or not. Might also just be the same. I'll keep an eye on it. Didn't clean any of it off, the node have only run for a day or two in total so far. But should I?

 

There is a systemp on the HWmonitor program and it stays around the 39-40degrees constantly. Guess that might be the one, but don't know where its placed on the motherboard.

 

All in all, yeah, I think you are still in the safe spot if your max load temps are just hitting 70 degrees. I think I would use whatever fan set up you feel most comfortable with and just run with it.

 

Pretty much sums up my thinking atm. Should spend all this time on getting assets sorted instead

 

Thanks again guys for all the great advice.

[Scott Schroeder]

I've always been under the impression that you should remove all thermal paste once you take the heat sync off. If you are doing a quick flip of the fan, then maybe not. But if you want to add more or leaving the old stuff exposed for a while, then you are probably going to be better off starting from scratch with the paste. That way the old stuff doesn't get in the way of the new stuff or get any dust and garbage from the air on it.

[joelmcwilliam]

Feeding the Noctuas enough cold air and let it seep out where possible, seems to work better than trying to remove the hot and not having enough cold air comming in.

Any recommendations on one good 120mm that could pull in lots of cold air but without sounding like a hair dryer?

 

 

Fans that can push a lot of air and still be silent. I think you should look at noctua or bequiet. Have a look at their site.

 

I can provide you with some links if you want more info about fans:

 

http://www.technic3d.com/review/kat86-luefter.htm

 

http://www.xtremehardware.com/cooling/air/

 

http://us.hardware.info/category/23/case-fans/reviews

http://nl.hardware.info/categorie/23/casefans/reviews

 

Another choice for a case would be a silverstone. Have a look at their site.

Edited May 4, 2016 by joelmcwilliam

[joelmcwilliam]

Very interesting thread!

 

I was wondering if it makes any difference building a workstation like that or going for a rack solution like the

Dell Rack Server R7610 2HE Intel Xeon 2x E5-2670 Ram 64GB.

 

It has about the same specs and is available at ebay for about €1000.

 

I'm just thinking that the rack solution is better for future expansion or adding some GPU's.

But I'm not really an expert there ;-)

 

Cheers.

 

-I don't see any connetions for graphics cards on that board.

 

-And even if it had, powerfull graphics card are quite big. Not sure how you want to make it fit.

 

-Besides that, the powersupply must be old (and die of age). And I'm not sure if it can actually deliver the needed juice when you would hook up some graphics cards to it.

 

-And the powersupply needs to have the connections to give the gpu's juice. Don't think the rack server power supply has the connections for gpu's.

[numerobis]

I was wondering if it makes any difference building a workstation like that or going for a rack solution ... I'm just thinking that the rack solution is better for future expansion or adding some GPU's.

.

Limited space, small fans = very loud, needs a rack...

[numerobis]

I would also remove the old paste even if it is only a few days old. And i would apply a bit more so that the whole heat spreader is covered. Are you sure that your contact pressure is high enough? A 2°C difference between the two CPUs is possible. CPUs are different. It could be much more.

 

You could use a second fan on the Noctuas which maybe could lower the temps 1-2°C. But i think the problem is the size of the cooler. They are much smaller than a D12 or D14. I think i would go for a single tower 140mm cooler like the U14S or Scythe Ashura instead even if they have to be mounted in a row. Maybe using 3 140mm fans.

 

edit: a 140mm cooler does not fit between the RAMs on a Z9PA-D8... a 120mm model like the Scythe Kotetsu is working.

 

How are your fans configured? Are they running full speed at 60-70°C?

Looking at the Fractal Design R2 120mm specs it has a max. airflow of only 69 m³/h which is quite low for a 120mm fan running at 1200rpm. Should be more like 100-110 m³/h.

 

Some cheap and good case fans could be the Arctic Cooling F12 PWM with up to 126 m³/h @1350rpm (~4€). For a more silent solution maybe the Noiseblocker NB-eLoop B12-3 (max. 121.2m³/h) or Fractal Design Venturi HF-12 (max. 141m³/h) (both ~13€).

 

You could try to undervolt the CPUs. There should be some headroom that can be reduced.

Edited May 27, 2016 by numerobis

[hkahk]

Limited space, small fans = very loud, needs a rack...

 

Thanks!

[hkahk]

You think a Corsair AX760 would be ok for tis dual xeon e5-2670 setup?

I don't need any GPU power.

There are quite cheap atm.

[numerobis]

should be more than ok. But cheap?

I'm planning to use my old Seasonic S12II 520W for this setup. I think i'll also build one or two E5-2670-sytems to replace some old sandy bridge nodes.

[hkahk]

should be more than ok. But cheap?

I'm planning to use my old Seasonic S12II 520W for this setup. I think i'll also build one or two E5-2670-sytems to replace some old sandy bridge nodes.

 

Thanks!

Well i have the chance to buy a couple of them used from some local dealer here.

I was just wondering because they habe 2x 4+4 pins insteas of 2x8 but I guess that's fine.

[numerobis]

4+4=8 EPS/ATX. The Seasonic has 1x 4+4 pin and 1x 8 pin. I think this should work - but i didn't try it so far, so no guaranty!

Theoretically these 115W CPUs should even work with 2x 4 pin.

 

edit: it is working.

Edited August 19, 2016 by numerobis

[hkahk]

Anyone knows if it really makes a difference for a rendernode to have the SR0KX instead of the SR0H8?

 

Thanks!

[numerobis]

Should be no difference, but why?

 

edit: it is possible that the new stepping is more efficient and needs less voltage (or can be undervolted a bit more)

btw. my parts should arrive tomorrow

Edited May 9, 2016 by numerobis

[hkahk]

Nice!

Thanks for the info.

I'll go for the SR0KX as well.

Did you get them from ebay? around 110€?

 

Best.

[numerobis]

Yes

[jensandersen]

I would also remove the old paste even if it is only a few days old. And i would apply a bit more so that the whole heat spreader is covered. Are you sure that your contact pressure is high enough? A 2°C difference between the two CPUs is possible. CPUs are different. It could be much more.

 

You could use a second fan on the Noctuas which maybe could lower the temps 1-2°C. But i think the problem is the size of the cooler. They are much smaller than a D12 or D14. I think i would go for a single tower 140mm cooler like the U14S or Scythe Ashura instead even if they have to be mounted in a row. Maybe using 3 140mm fans.

 

How are your fans configured? Are they running full speed at 60-70°C?

 

You could try to undervolt the CPUs. There should be some headroom that can be reduced.

 

 

Noted on the thermalpaste, will redo both of them soon. Thanks for the case fan examples, sadly I just ordered a single 120mm Noctua case fan (NF-S12A PWM at around 107m³/h). It will replace the right side single Fractal one for either intake or exhaust. Probably exhaust as I'm planning to revert to my org. air flow setup and redo the thermal paste at the same time.

 

Thinking the two left side Fractal 120mm are sufficient in providing air straight to the CPU Noctuas, and then having the new 120mm Noctua case fan with much better airflow, being able to exhaust all that heat. I'll remove the two 80mms again, as they just seem to dirupt the flow.

 

Atm. the temps are with the fancontroler set on medium, so I'm guessing its only around 8-900rpm on the Fractal fans.

 

The case/node i've chosen has a max height for CPU coolers at around 13cm, so these were the best ones I could fit. Again, I know its not an optimal case performance wise, but I chose it for the form factor, stackability and looks.

 

Nice to see others wanting to try a build with the 2670s, best of luck and let us know how it goes

[unclefarkus]

Do you have any tips on buying from ebay? I see many. I've never purchased used computer components, so I'm a bit nervous. But the power sounds amazing for the price.

[jensandersen]

Hi Adam

 

Was the first time I did so too. I just looked for a seller with good reputation and where the item description looked good and not lacking/hastefully written. Also make sure that your seller ships from inside the US (in my case EU) so you won't get any annoying customs tax on top of the price + long shipping time and cost.

 

I think eBay can help you with a refund if the item is faulty, but make sure so check the seller's "return policy". All in all, it was a pretty good experience and the item came quickly+well packed, but I guess thats really up to the seller.

[Matt Sugden]

Jens, how has this build worked out for you in the end? I was contemplating doing the same, but I need to build 6! So I'm a bit nervous about using second hand discontinued chips. Also the mother boards for these seem not easy to get hold of. Did you manage to sort out your cooling issues? I'd be sticking these in rack units which are lower profile than the case you used so I'm a bit unsure about cooling as those big heat sink you used wouldn't fit.

[jensandersen]

Hi Matt

 

I'm very happy with the build. There is always a risk when buying second hand items. That is part of the price, but the value of these chips for the current prices are second to none. I'm not sure on the failure rate of xeon cpus, but I can't imagine they are close to anything worth worrying about. They are meant for server/heavy workstation use after all. So if you are able to find trustable people on ebay, I'd go for it.

 

What racks are you using? The motherboard I chose had to fit in to a normal atx case. You might have more options when going with a rack build.

 

Check the CPU clearance level of the rack. Noctua provides many different cooler sizes. My choise is actually meant for 3U racks.

 

Temperature wise I don't worry about it any more. The "conclusion" from the great people through this thread is, that I'm okay as is. That said, I haven't had the time yet to mount the new case fan, reapply the thermalpaste and try a 3rd airflow approach. I'm thinking it could result in a reduction of a few degrees. Might not be worth the trouble, but I find it fun to tinker with.

 

Depending on where you'll have the racks placed, you could just have all fans go 100% all the time and probably see much better temps. My node is right behind me on the shelves, so that's not something I do (due to the noise level).

 

Good luck, please feel free to contact me if you have any other concerns/questions.

[Matt Sugden]

Thanks for the detailed reply Jens, I think getting the right cooling for the processor is going to be the key to the success of this build. I need to find a fan that has good cooling performance, is compact and also quiet. I guess the holy grail of fans!!

 

My rack cases are only 2U high, not dissimilar to this https://www.xcase.co.uk/2u-rackmount-server-cases/x-case-x255f-standard-atx-2u-550mm.html

 

They currently have 3 fans pushing air across the board from the front, plus the CPU fan and the PSU fan. Individually they are not too noisy, but obviously this multiplies as all 6 are switched on.

 

I was thinking something similar to the Dynatron R24 2U Active CPU Cooler, but I'm just not sure how well a 2u rack will cope with the heat of two processors.

[gudrungeest]

Hi Jens,

thank you for your great description. I followed your example exactly and this enabled me to build my first computer.

It came out as a fantastic rendernode. No problems with the cooling either.

Happy greetings from Hamburg

Gudrun

[xjapanese]

Hi Jens,

 

Thanks for the detailed sharing of the build.In case you are using vray can u share any times benchmarks compared to your 5820k station or any of you guys that made the same build .

 

Thanks

[RyderSK]

Cinebench R15

 

i7 4930k = 950 ( turbo ) / 1200 ( extreme 4.8 overclock )

2x2670v1 =2000

2x2680v2 =2660

2x2698v4 =4600 (ES)

 

CoronaBenchmark (imho not as accurate because too short, my DR rendering is more accurate)

 

i7 4930k = 210 seconds

2x2670v1 =96 seconds

2x2680v2 =76 seconds

2x2698v4 =40 seconds

[jensandersen]

Hi Jens,

 

Thanks for the detailed sharing of the build.In case you are using vray can u share any times benchmarks compared to your 5820k station or any of you guys that made the same build .

 

Thanks

 

You're welcome. You can rest assure that Juraj is spot on in his answer (Thanks Juraj).

 

 

Hi Jens,

thank you for your great description. I followed your example exactly and this enabled me to build my first computer.

It came out as a fantastic rendernode. No problems with the cooling either.

Happy greetings from Hamburg

Gudrun

 

 

Hi Gudrun, I'm glad it was helpful to you! Enjoy your new node

Edited August 19, 2016 by jensandersen