Cheap 16core Xeon Node - Pros/Cons?

[jensandersen]

**You can find a more structured post about building this render node on my blog HERE**

 

*Updated post (below) with what I ended up buying, pictures of the build and some quick tests*

 

Hi guys

 

I came across this article on techspot: http://www.techspot.com/review/1155-affordable-dual-xeon-pc/ and was hooked! Sounds like great value vs price.

 

Currently I only have my 5820k workstation and would love to have a single node to push work to so I can continue using my workstation.

 

The 2x Xeon E5-2670 C2's are on their way from eBay now and the rest of the hardware I'm about to purchase is this:

 

MB: ASUS Z9PA-D8 (ATX form factor)

 

Coolers: 2x Noctua NH-D9DX i4 3U

 

CASE: Fractal Design Node 605

 

RAM: 32gb total (maybe 64gb) Samsung - DDR3 - 8 GB - DIMM 240-pin - 1600 MHz / PC3-12800 - CL11 - 1.5 V - ECC

or

Kingston ValueRAM Server Premier - DDR3 - 8 GB - DIMM 240-pin - 1600 MHz / PC3-12800 - CL11 - 1.5 V - ECC - not 100% sure if they will work, but they are dirt cheap. All "supported" ram on the board's support page from Asus states ECC is needed

 

PSU: eVGA SuperNOVA Gold 650 G2 - I'd like a platinum one, but if I need 2x 8pins for the 2xCPU, they get both really expensive and 700W+ and that seems to be overkill. Thought of splashing for a Sea Sonic Platinum Series 520 Fanless, but this model does not have 2x8pins. The dealer would not recommend a molex->8pin converter, any experts here?

 

GPU: Not needed.

 

Storage: 120gb ssd (already purchased).

 

The case might be a bit small for dual xeons, but I really like the form factor of it and it would fit perfectly in the home office.

 

Anything I have missed or should reconsider?

 

Would it be stupid to also use the node as a server/NAS for work assets, backups etc? Everything is in the workstation atm.

Edited September 14, 2016 by jensandersen

[Scott Schroeder]

I would look at the possibility of water cooling those things, especially if you think the case is a bit small for dual xeons. I would imagine, cranked at full render, those things are going to pump out some heat.

 

Is that motherboard compatible with that case? PCPartPicker.com doesn't show it being compatible or compatible with dual CPU motherboards from a quick initial search. That may or may not be true, but you don't want to have to deal with space issues if it "just fits". I would probably go with something bigger and allow better air flow if you are going to go fan cooling.

 

I'd up your RAM to 64. If you have dual CPU's, get 2x's the RAM.

[jensandersen]

hmm, the motherboard is listed as a normal ATX and thats what the case say it can fit. Watched some youtube vids of the case and I think it could work. Its possible to mount 3 120mm intake fans and 2 80mms out, so without a GPU and just a single SSD, I think the airflow should suffice for the noctuas.

 

Thx for the heads up on ram.

 

Any idea if the 520W fanless PSU would suffice?

Edited April 13, 2016 by jensandersen

[Scott Schroeder]

Size wise, it does fit the case. I'd be looking at heat and getting that hot air out of there. Don't forget about all of your cables and such running on the inside of the case and how those could restrict airflow. Some more research is probably warranted to make sure you protect the investment. I know when I built my machine, my case and keeping it cool were to 2 areas I spent most time researching.

 

I would be worried that using a fanless power supply for this sort of case and the loads it would be under. This is a rendering box and will be under heavy loads for extended periods of time. I'm not familiar with how robust fanless PSU's are in terms of being used in render boxes.

 

The original article recommend at least 600w PSU. Though you aren't using a GPU, so perhaps you could use a lower wattage PSU. I used this, http://outervision.com/power-supply-calculator, and without a GPU you come in at around 320W of usage. Granted, I have no idea how accurate that site is.

[joelmcwilliam]

You might find some info I posted earlier here usefull: http://forums.cgarchitect.com/80055-3d-rendering-workstation-around-6000-euro-dual-monitor-setup-included.html

 

If you are uncertain about the motherboard fitting in the case you can send an e-mail fractal design. I don't have nuff time to google.

 

Don't worry about getting an overkill psu. It won't hurt and if you later on decide to plug in some extra gpu's in the motherboard or use the psu in an other build the juice is there. Flat cables in such small case would be preferable. Like scott said, no need to go for a fanless. Heat is a killer. Good silent psu's with a fan are out there and you won't be able to hear them. Have a look at the site of Super Flower (or Bequiet or Corsair).

 

Wether you want to use the node for a server or nas, I think that will be a question that you alone are the best to answer.

[jensandersen]

Thanks for the replies guys.

 

After further reseach, I've ordered the parts above.

 

Opted for 64gb and a EVGA Supernova G2 650W 80 Gold PSU. It got a perfect review at http://www.jonnyguru.com/modules.php?name=NDReviews&op=Story2&reid=429 + it has two 8pin CPU connectors for the Xeon board and a fan!

 

Case will end up having 3x120mm fans for intake and 2x80mm for outtake. My gut feeling is, that together with the Noctua coolers and no GPU installed, it should be okay temperature wise. Time will tell.

 

I'll post how things goes when I've received the parts next week and done some tests.

 

Wether you want to use the node for a server or nas, I think that will be a question that you alone are the best to answer.

 

It was more a technical question as to wether the node would be effective also as a server/nas while rendering. If transfer speeds, backups etc would halt to a complete stop or slow down alot, it might not be a good idea.

[jensandersen]

***UPDATE***

 

So, got the parts and everything fit nicely and it runs great! (Noctua coolers had me holding my breath for a second though).

 

Case opened up. Removed the HD bays for easier access:

 

Installed extra fans:

 

Mounted the Noctua coolers (thight fit!)

 

The 2x80mm fans position wasn't good for the airflow. So I removed them to let the 2x120mm fans provide air in to the case, the Noctuas now pushes the cold air towards the other side of the case, where it gets sucked out by the 120mm I flipped.

 

I removed the dust filter on the outtake 120mm and used the internal fan cover on the outside instead. This was much coarser and my thinking was it was easier to move air through.

 

Everything fitted (I removed one of the front I/O pieces as I don't need the ports and it would allow for cleaner cable management inside the case)

 

Rather simple case, but I like it. Fits perfectly on the shelves. Might even be space for having another one stacked on top of it.

 

Temperatures.

The 3x 120mm case fans are connected to the case's fan control switch and this test was done with it on the lowest setting. At this setting and on idle, I can't hear it at all. Even though its right behind me. After a few minutes on full load, the Noctuas are hearable, but very acceptable.

 

Scene rendered with Corona and the temperatues was stable after like 10-15mins. Much better than I expected. Will have to see on multiple hours of renders. But off to a great start!

 

Performance:

The node is nearly twice as fast as my OC'ed 5820k workstation. The new Corona 1.3 benchmark test was cleared in around 1min41sec. My workstation took around 3min.

 

So all in all, I'm very happy with this build. The price vs value is absolutely great and I didn't have to get a giant tower for a case.

Edited August 19, 2016 by jensandersen

[joelmcwilliam]

That looks sweet and so clean. And silent too. Good job man! Thanks for letting us know how it turned out.

 

How is the powerconsumption? (only if you have something to measure)

[Scott Schroeder]

Very nice build! Glad to see all of the components fit together. How's the heat levels now that you've had it for a bit?

[Nikolaos M]

I wouldn't feel so comfortable with these temps (idle ~40° and max ~70°). I expected much lower temps for 115W tdp Xeons with a max 3.3GHz frequency, to be honest.

Edited April 26, 2016 by nikolaosm

[andpic]

Hey man, great build!

 

Just a few questions:

 

. how much? total?

. what do you have there in terms of software? And how do you use it in terms of workflow? Sorry I'm a bit newb in this area...

 

thanks!

[jensandersen]

Very nice build! Glad to see all of the components fit together. How's the heat levels now that you've had it for a bit?

 

Had it running for a few hours today, this time with fan control on medium (instead of low). At this level the case fans matched the Noctuas noise output, so no difference in noise level, but temperatures dropped to around 65°. So a ~5° improvement.

 

I wouldn't feel so comfortable with these temps (idle ~40° and max ~70°). I expected much lower temps for 115W tdp Xeons with a max 3.3GHz frequency, to be honest.

 

Hmm, this is not something I was hoping to hear. Was under the impression that those temps were okay. What levels should I have hoped for to be "in the clear"? Could this be too much thermal paste? I just put a small dot in the middle of the CPU, little bit bigger than the manual said though.

 

Does the fact that these are v2 and not v3 Xeons have anything to say? You are making me nervous here Nikolaos!

[Nikolaos M]

Hmm, this is not something I was hoping to hear. Was under the impression that those temps were okay. What levels should I have hoped for to be "in the clear"? Could this be too much thermal paste? I just put a small dot in the middle of the CPU, little bit bigger than the manual said though.

 

Does the fact that these are v2 and not v3 Xeons have anything to say? You are making me nervous here Nikolaos!

 

It wasn't my intention to make you nervous, Jens. Not at all!

Being well informed about your new render node's abilities and limits will help you to keep out of potential problems with it.

 

The 5° C improvement you noticed could be an expectable improvement over the days, and not just a fan speed adjustment result. Thermal pastes usually need a couple of days of operation to spread out evenly and settle to their optimal state for maximum performance. Of course, it might be the result of lower ambient temps too, so I couldn't be 100% sure. The last two days here in Greece we saw a 5-10°C decrease in temperatures. I don't know how the weather changed the last 2 days in your country.

 

I would personally expect temperatures like 25-30° for idle and ~50-55° under full load at this time of the year. It's just an estimation based on experience and nothing more. I could be wrong and hope to be wrong in this case. Besides, I now see a TCase max temp of 80° for these Xeons inside Intel's site, so there should be a good headroom in any case and you shouldn't worry so much imo.

 

I would do a little research, though, about the normal operating temperatures of this specific Xeon, or others of its processor family. If the feedback shows that your temps are in deed higher than normal, then 3 things could be wrong imo:

1. Bad installation of the coolers or improper amount of thermal paste (less, or more than needed),

2. Poor airflow inside the case. This case might fit everything inside and be full of extra fans, but that doesn't mean it's the best performer in this type of operation (to tell you the truth, I wouldn't trust a htpc-like chassis for this job), and

3. The motherboard itself. The two sockets of the D8 are so close that the Noctua coolers practically touch each other. It wouldn't be my first pick for a render node, and that's the main reason why.

 

I know, my post comes now that everything is set and working, and few things can be changed, but it's better to have a clear view of the situation from the start and not be surprised in the future if something goes wrong.

 

PS Could you post your temps with another temp software, like Real Temp or Core Temp?

Edited April 27, 2016 by nikolaosm

[Scott Schroeder]

I don't know, 65-70 degrees for fan cooling at render time doesn't seem to be all too bad. My old early gen i7 would render at 95 degrees with the old stock intel fan and with an upgraded fan it would still peak out at around 75 degrees. That old i7 is still working almost 8 years after I got it and has thousands of render hours on it.

 

You may want to rethink your fan flow. I think you mentioned you have 2 different areas of air coming into the case? From my understanding, if you have 3 fan areas, you'd want 2 to be exhaust and only 1 to push in fresh air. You want to create almost a wind tunnel.

 

You may have to think about investing in water cooling to make the case really work.

[jensandersen]

Thanks for the replies guys, I'm a bit more at ease now, but will definately keep an eye on the temps. I read an article (before purchasing) over at Puget:

https://www.pugetsystems.com/labs/articles/Impact-of-Temperature-on-Intel-CPU-Performance-606/ and here they mention a limit of 100° - that seems a bit extreme though.

 

From my understanding, if you have 3 fan areas, you'd want 2 to be exhaust and only 1 to push in fresh air. You want to create almost a wind tunnel.

 

Yeah, that was what I aimed for. This is my setup (2x120mm taking air in and 1x120mm pulling it out):

[Scott Schroeder]

I wonder if you reverse the flow. That way the hot air from the CPU doesn't linger in the case before it gets pushed out. The front fan should be enough to push fresh air in and let the dual fans in the back exhaust. Especially since you don't have to worry about GPU heat from this set up.

[joelmcwilliam]

Limit of 100° isn't that extreme. A few years ago intel mentioned a Tj Max with every chip on their site. They were for some high end chips around 105 degrees celsius.

 

If the temp gets towards that point it will start thermal throttling. That means the chip will clock itself lower too cool down and will clock itself higher when its cool enough. Lower clockspeeds will get you lesser performance.

 

Now I see intel doesn't mentioned a Tj Max, but they mention a TCASE .

 

Scott has definately a good point with reversing the airflow.

[jensandersen]

Lovely with this feedback guys. Thank you. I'll try and reverse the fans this weekend, should be a quick fix (unless I have to dismount one of the Noctuas to be able to flip those fans).

 

Would it be worth it to put one or two of the 80mm fans back in? They would be placed just "below" the Noctuas on the image and set to exhaust. Or would that just make it worse? Hate having bought them for this system and now they are in the drawer...

[Scott Schroeder]

It might make it worse as it would want to draw some of the hot air from the top most CPU across the bottom one then out the case. I think the two in the back should be enough to handle each CPU's fan exhaust.

[joelmcwilliam]

I was wondering if you could do some real world performance testing in the software you use. (not with benchmark tools)

[jensandersen]

Right, so flipped the fans and also had to dismount one Noctua to get to their fans. Is this an okay look for the thermal paste?

Thought it looked kind of scarce and weirdly spread, so added just a tiiiny bit more to just this one.

 

Sadly, I see no difference in temperature with this reversed setup. Might even perform 2-3degrees worse, but that could also be the weather outside starting to heat up.

 

 

 

I was wondering if you could do some real world performance testing in the software you use. (not with benchmark tools)

 

The Corona 1.3 benchmark is "real world". It loads a scene as in 3ds max and renders it. It's the same result I get when rendering my normal scenes as with this one. The node is still roughly twice as fast as my 5820k in the workstation. FYI: The 5820k is OC'ed to around 3.8. Had it running around 4.1ghz, but crashed a few times and havn't bothered spending more time fine tuning it.

[joelmcwilliam]

Thermal paste spread is just fine. It is nicely spread. And yes, you could put an tiny amount extra to get a wider spread. Thermal paste is just needed to fill the gaps you see in the metal when you would put it under a microscope. You can even spread it thinly with your fingers if that is what you want to. No worry's here.

 

Too bad the reversed airflow din't make a improvement. Guess you will be going back to the original airflow setup. Why don't you put the two smaller fans you have laying around back in the chassis? Where they originally were mounted, making the hot air go out of the case as quick as possible.

[hkahk]

Very interesting thread!

 

I was wondering if it makes any difference building a workstation like that or going for a rack solution like the

Dell Rack Server R7610 2HE Intel Xeon 2x E5-2670 Ram 64GB.

 

It has about the same specs and is available at ebay for about €1000.

 

I'm just thinking that the rack solution is better for future expansion or adding some GPU's.

But I'm not really an expert there ;-)

 

Cheers.

Edited May 4, 2016 by hkahk

[jensandersen]

Too bad the reversed airflow din't make a improvement. Guess you will be going back to the original airflow setup. Why don't you put the two smaller fans you have laying around back in the chassis? Where they originally were mounted, making the hot air go out of the case as quick as possible.

 

The two 80mm fans was some I bought on the side. I'll try and mount them again for the extra exhaust, see what happens.

 

I'm thinking that the two 120mm fans for exhaust can't keep up with the Noctua's in this reversed setup. So the hot air gets "trapped" inbetween the CPU coolers and the exhaust fans. As when I put the fan control from medium to max, temperatures drop 3-5degrees pretty fast. Makes it sound like a table fan next to me, but nice to see better temps are possible when it needs to run for an extended period of time.

 

Will definately have to try putting in the 2x80mm fans, checking the other CPUs thermal paste spread (as they differ almost 2degrees from each other, but that might just be placement in the case) and then back to the old airflow and see.

 

In any case, I think I'm safe with temperatures just below 70degrees for these Xeons under load. Can't imagine they would run any cooler in a rack system.

 

I was wondering if it makes any difference building a workstation like that or going for a rack solution like the

Dell Rack Server R7610 2HE Intel Xeon 2x E5-2670 Ram 64GB.

 

I don't know much about rack systems, but think they are intended to be stored in a rack mount (rather large thing) and also kept in a server room that has either great airflow or is cooled. Those rack soloutions have a lot less space inside than the case I've chosen and I think noise/temperatures would be an issue if you were to use it right next to you.

Edited May 4, 2016 by jensandersen

[Scott Schroeder]

It might have less to do with the 2 fans in the back than the single fan in the front not keeping up in pushing air in. I had hoped that it would be enough to push cold air in, but maybe it's not. The Noctua's may not have enough cold air to cool, even though the exhaust is getting pushed right off the Noctua's and right out of the case. When increasing the back 2 fans speed, then you can start to draw more cooler air in from the front and other vents.

 

Be careful with the thermal paste. Too much of it can cause high temps as well. You are cleaning off the old thermal paste when applying new stuff, right?

 

Can you monitor your ambient temps in the case, not the CPU temps? I can monitor temps on all my components from mother board to hard drive, etc with my corsair set up so I can tell if there is more hot air just lingering in the case.

 

All in all, yeah, I think you are still in the safe spot if your max load temps are just hitting 70 degrees. I think I would use whatever fan set up you feel most comfortable with and just run with it.