Multi CPU motherboard question

[branskyj]

Hi all,

I have been having dirty thoughts lately of purchasing 4 socket CPU motherboard with some of the latest Xeons LGA 2011 socket and have several questions regarding that madness:

1. How do I know that I can run 4 CPUs on a board? What is the parameter I should be looking for? I don't want to get 4 CPU only to realize they can run in pairs only.

2. Can I (in the beginning at least) purchase just a single Xeon and then gradually increase their number to 4? In other words- can I install one CPU and RAM to a single slot on the motherboard and expect to have a fully functional PC or must I have all the slots on the board populated with CPUs and RAM?

3. If the motherboard supports LGA 2011 socket does it mean it will support the latest and greatest LGA 2011 Xeons or should I download the motherboard manual and look it up.

 

I have a very descent set-up at the moment (3930k Workstation and 2700k render node) but that have it's disadvantages as well as advantages.

 

Cheers for all the help, guys.

[RyderSK]

If you don't know the answers to this, maybe building server-based workstation isn't such a good idea ;- ).

 

With that, Quad-socket xeons aren't currently out yet at all and won't be atleast for few months I think.

Nonetheless, they won't be affordable. Keyword here is space effieciency, not raw power.

 

http://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors#Xeon_E5-4xxx_v2_.28quad-processor.29

 

Dual sockets are available, and decently good deal.

 

http://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors#Xeon_E5-2xxx_v2_.28dual-processor.29

 

Because we want them to work as powerful workstation/render station alike, it's really important to buy the correct ones, ie. the highly clocked 8 and 10 cores, not the low-clocked versions, even if they have higher core count (12). Some people have this weird idea that more cores is always better, and then they buy 2Ghz clocked weaklings ! I would laught, but these are some otherwise knowledgeable 3D folks...

 

The afforable workstation boards lg2011 are from Asus and Micro alike, with the Asus probably being rather better choice, although slightly more expensive. E5 26xx v2 are supported with UEFI(Bios) update.

 

You can buy a single Xeon for start, but take in mind the memory modules are allocated per CPU, and even then, it might be hassle. It's not advised.

 

I think decent price point where dual-socket workstation based on e5 26xx v2 start to make sense is around 6000+ euros (at this price, you roughly get performance of clocked pair of 4930k nodes). So it's not the price you save, but you end up with single powerful workstation, which I think is always better choice and the price difference is eventually negligible in grand scheme. Of course, those who enjoy servising 10 little boxes around...everyone's choice.

 

Maybe if you want to see how such workstation would look like, go to BOXX's website, look for some above 40 000 euros computer, than divide that number by 6, and you can build identical setup = ) Just copy the parts (apart from ugly Case, that one is custom)

[Dimitris Tolios]

Well, a 4P system can be great if you run your own math models and need to stick with x86. The G34 AMD platform works wonders with that.

Issue is, desktop versions of Windows are limited to 2P systems, you will need some Windows Server edition to actually "see" more than 2 processors. This is a "killer" as far as economy goes, as the cost for the OS alone will surpass that of a 2nd 2P motherboard and a 2nd PSU, making it a "space issue" more than a power issue - as Juraj already wrote: you just buy more compute density with 4P, you are not really saving any money over networking more 2P nodes.

 

Large render farms care for that, but I doubt it is really an issue for home/small office based installations, unless again we are talking deploying that many machines or having that limited of space where an extra headless box or two become a hassle.

 

And ofc you will need to drop serious money for any Xeon to just approach the single threaded performance you get from a stock 3930K and maintain a small multithreaded advantage over the 6C/12T i7s, and you will never get the performance you get with a stock 4770 regardless of cost on single threaded, as current Xeons are a generation behind the desktop models, and running a tad lower clocks at that. Ofc with overclocking - if it is even in the picture - i7s are un-catchable.

And when up-to-i7 levels single threaded performance is possible, we are always talking Xeons that are good for up to 2P systems: the 4P capable E5s are limited below 3.3GHz for the 8 core parts, and below 2.8GHz for the

Edited January 20, 2014 by dtolios

[branskyj]

First of all- big apologies for not replying sooner, I was given some dreadful deadlines at work and couldn't write back.

 

My plan was to get 4 of the E5-2697 V2 (not all at once but gradually) and find before that a motherboard that can actually house all of them. Now that I know 4way with these is not an option currently that changes things drastically. 2 of them will be very good but if I have to wait for the 4way motherboards I will wait.

 

"The memory modules are allocated per CPU, and even then, it might be hassle"- could you please elaborate, is it because the Xeons require ECC Ram which is more expansive or...?

 

I was not aware regular Windows versions are limited to 2 physical CPUs, I just checked the price for Windows Server 2008 R2 and that bad boy will set me back around 510.00 GBP. That is indeed a lot but manageable since it's a one time investment.

 

The single threaded performance is of no concern to me since the workhorse will be used predominantly for 3D visualization of massive scenes and 3DS Max and VRay are heavily multi- threaded. As for the E5-2697 V2 vs 3930K- a single Xeon is roughly 37% faster than the 3930K according to Toms Hardware benchmark which means that 4 of the E5-2697 V2 will roughly equal 5 3930K. I don't know about you guys but I am drooling here . I am not even interested in overclocking when it comes to 3D work by the way.

 

Again- thanks a lot for all that precious info, guys. Some good news and some bad news there I guess.

Have a good weekend.

[Dimitris Tolios]

The single threaded performance is of no concern to me since the workhorse will be used predominantly for 3D visualization of massive scenes and 3DS Max and VRay are heavily multi- threaded. As for the E5-2697 V2 vs 3930K- a single Xeon is roughly 37% faster than the 3930K according to Toms Hardware benchmark which means that 4 of the E5-2697 V2 will roughly equal 5 3930K. I don't know about you guys but I am drooling here . I am not even interested in overclocking when it comes to 3D work by the way.

 

 

Well...3DS is not heavily multi-threaded, Vray is...the modeling process, even the material preview won't give you any real edge using a slower per-core Xeon with more cores, the contrary.

 

A single E5-2697 V2 is roughly $2,750.

You should be able to build a decent 4930K based render node, with basic GPU, a decent mobo and PSU, 32GB RAM, a 120 class SSD and OS for roughly half that. For the price of the 4 E5-2697 V2s, the board and the Server OS, we are talking around 10x such nodes. Not even started talking ECC Ram etc.

 

The GHz aggregate in each case (we are talking IB-E architecture in all, so it is kinda fair):

 

10x 4930K CPUs = 3.4GHz * 12 Threads / CPU * 10 CPUs = 408 GHz

4x E5-2697 V2 CPUs = 2.7GHz * 24 Threads / CPU * 4 CPUs = 259 GHz

 

So yeah, amazing for the density (10x nodes take a lot more space than even the huge case that 4P will require), but not very cost effective.

 

It is very effective in micromanaging it tho. No coordinating 9-10 nodes (tho with identical hardware, a single image of the 1st complete node can be transferred to the others within minutes), less heat etc.

 

Long story short, most 4P/2Ps won't be really better workstations than the average 4930 or even 4770 i7, and won't be able to render faster per $ spent than multiple render nodes.

 

"The memory modules are allocated per CPU, and even then, it might be

hassle"- could you please elaborate

 

Each CPU has its own memory controller and memory bank. Say for argument shake you would go for 32GB system memory. That's 32GB/4 = 8GB per CPU. It is not exactly segregated, but it is not really possible for CPU #1 to access data in CPU #2's memory bank seamlessly. "Seamless segregation" is a big think in the R&D for multi-CPU compute platforms.

[branskyj]

Thanks for the quick response Dimitris. You are always a big help.

 

I don't know why I said 3DS Max is multi-threaded when I actually knew it's not :)Thank God for Max 2014. I meant VRay is multi-threaded.

Building a 4930K workstation would only give me a slight advantage over my current setup and at the end of the day I am still stuck with a single CPU workstation no matter what. And one day I will find myself wanting more power again...

 

I am just fed up with render nodes- I had two before, I have one now and even though they are very helpful you always get problems(at home and at work). I am running my PCs on Samsung SSDs and all textures are stored locally on each PC but I still get delays before the rendering node kicks in, missing textures even when they are all there, more power consumption, I need to keep the node connected to one of the monitors to oversee what is going on (VPN clients are even worse)

 

I still don't understand your explanation regarding the seamless memory access. A 3D scene and all it's textures are loaded into the RAM. So are you saying that it has to be loaded into each CPU's RAM and then there might be issues when accessing the data from all the memory banks?

 

Again- thank you for the quick response.

Cheers.

[Dimitris Tolios]

There isn't a benefit to go to another system if you already have a 3930K...that's true. After Sandybridge, the lack of real competition on the AMD side as far as absolute performance CPUs, and the software actually being slow to pick up (things are usually sub-optimized), there hasn't been a real breakthrough in either absolute IPC performance or clock speeds (at least stock).

 

Even if I would get a 4P, I would strongly consider keeping it as a rendering node, but ofc it depends on the ratio of rendering samples vs. modeling, simulating particles etc (i.e. multithreaded vs. single threaded) is in your case.

 

If all you need is faster previewing, maybe a combination of additional CPU nodes and GPU accelerated activeshade viewports would be the way to go, as it is true that nodes do lag. I have no idea how easy it would be setting up a 10GBit network - if we are talking just a few machines, it could be a good option. You can get 10GBit NICs pretty cheap on ebay these days.

 

Rendering is memory intensive, and it is true that the more the threads information has to be spreaded on, the more memory you need, as each thread needs access to its own lil memory pool, and parts of the same information gets duplicated in each one of them.

Some people recommend 2-3GB per thread available...you realize that in a 4P system with 96 threads, this might be a considerable expense.

 

Many people end up disabling hyper-threading completely, so that more memory per thread is available without breaking the bank.

Depending on the task, the performance hit might be minimal, or it might actually get much faster when memory really becomes the bottleneck.

In real life, hyper-threading is not even close to double the performance as the simplistic aggregate GHz calculation I've used above might suggest, but in most cases it does help, in a absolute % that is application dependent or even varies between different files, scenes or even frames of the same sequence.

[RyderSK]

Julian, if you check that wikipedia page, you will see the 4-socket CPUs will not be available in powerful clocked version like the 2697 E2 and they might possibly be more than 50perc. expensive per unit as well. I would really forget about the 4unit node. There is also big chance that no E-ATX (for example the Micro is even regular ATX and the Asus is E-ATX sized SSI CEB board so with some "handy work" it can be fitted into regular XL Case) will be made available for this platform since under no case is it by anyone considered workstation material. And I would highly suggest not building a 4U rack case.

 

I though, completely agree with the logic of buying 2-socket top-range xeons. Managing 10-nodes, while cost effective, is pure craziness... I hate these budget Ikea solutions.

 

Well of course you get problems when you store all-things locally, I think investing in good NAS is right now the best investment to do. I use Synology with link aggregation and I get 160MB! transfer rates through switch with writting speeds, and this is using regular Western Red(7200rpm) disks.

 

I value comfort much more than price effectivness. My nerves and time are more valuable. But dreaming of 4-socket xeons on currect architecture is building castles in the sky :- )

 

I have no idea how easy it would be setting up a 10GBit network - if we are talking just a few machines, it could be a good option. You can get 10GBit NICs pretty cheap on ebay these days.

 

I'll tell you this summer :- ) I plan on ordering some refurbished Dell switch and the cards are ok price. In meantime I set-up link aggregation (Asus x-79 deluxe has double 1gbit ports) through workstation to NAS and man..even without SSD it's just nice. But once going 10gbit and SSDs in file server, I can't even image...that will be heaven.

Edited January 23, 2014 by RyderSK

[branskyj]

That's all good stuff, Juraj. I guess I will have to embrace the 2 Xeon route myself and just accommodate my workflow to it. Besides the more I think about the Windows Server edition needed to manage >2 Cpus the more I dislike the idea.

I wish you good lick in your hardware endeavours.

 

Best wishes.

[RyderSK]

There is also last possible caveat, that Vray would not scale properly. There were lot of people who were burned by buying cheap quad-Opterons by AMD only to find the architecture, while superb for folding and various multithreaded math calculations, did completely fail in rendering for some odd reasons.

Dual-Xeon is known to scale fine in Vray (but for example, some other rendering solutions have problem during some parts of calculations, rendering is not so properly multi-threaded in all parts) but there is possiblity that would be not the case in 4-socket. And to find out after buying such machine, that it in fact is useless for our purpose, would be unlucky :- )

 

But that is only wild guess, but just stating the possibility.

[Dimitris Tolios]

There are also 2 more CPUs that in my opinion are worthy of consideration: the 8-core E5-2687W V2 (150W) and the "vanilla" E5-2667 V2 (130W).

 

The W is a 3.4GHz base / 4.0 GHz Turbo 8 core, and the E5-2667 is a 3.3GHz Base / 4.0 GHz Turbo.

 

That is a healthy 25% boost in clocks across the board over the fastest 12C/24T, the E5-2697V2. The Turboboost V2 supported by all IB-E chips, allows more than one core to boost simultaneously, so the benefits can be maintained throughout the band of frequencies.

 

Single threaded or lightly threaded operations (or multitasking with single threaded apps) will benefit quite a bit. These clocks technically bring them up to 4930K/4960X speeds, while maintaining a 2C/4T advantage per CPU over the i7s. Such a 2P would still be "in the ballpark" of 2.5+ times faster than a i7 49xx workstation when rendering.

 

Versus the 12C E5-2697, both high-speed 8Cs will get a healthy ~25% single thread advantage, and overall lose 20-23% in the multi-core aggregate (theoretical, based on base speeds). Same aggregate calc puts it 2.56 times faster than a 4930K.

 

Also important is that the 150W 2687W is already $500 cheaper than the top 12C. That's $1K for the pair. The 2667 (which probably would by my choice as it only "gives" 100MHz base frequency, but still boosts to 4GHz) should be even cheaper. We are still talking $2050 ea. or so, but that is $1400 less for the pair.

 

Choices, choices

Edited January 23, 2014 by dtolios

[calinlihet]

Hello all, in my search for answers and just surfing the web at work I stumbled across this thread and a thing I can tell for sure is that I couldn't get more that half the rendering speed on 2 E5-2687W over a regular, non overclocked I7 3930K, I tried every solution, but vray, or max, just get confused over 32 cores on one machine, sure, DR works a lot better now with 2 beasts but I wasn't worth the money, if I'd knew I would just have bought a couple more 3930k or 4930k. IMO.

[Dimitris Tolios]

Hello all, in my search for answers and just surfing the web at work I stumbled across this thread and a thing I can tell for sure is that I couldn't get more that half the rendering speed on 2 E5-2687W over a regular, non overclocked I7 3930K, I tried every solution, but vray, or max, just get confused over 32 cores on one machine, sure, DR works a lot better now with 2 beasts but I wasn't worth the money, if I'd knew I would just have bought a couple more 3930k or 4930k. IMO.

 

So, the 2P is 50% slower or 1.5x faster than the 3930K ?

Kinda missed you there.

 

Regardless of what might be wrong (cause if it is slower, something is wrong), 2P systems were never really a "high value for $ spent" proposition.

[calinlihet]

Oh, sorry. English is not my first language. The 2p are twice as fast as the 3930k but only. I expected something faster. I mean 12 logical processors vs 32 at almost the same clock speed. 3.2ghz on the 3930k and 3.1 on the xeon.

[branskyj]

Hi Calin,

I am not sure why would you expect anything more than twice the speed of the 3930K. In fact I think you should feel lucky you are getting that much. I know one can overclock their 3930K and reduce the difference but I never liked the idea of overclocking of a workstation. It's easy and sometimes perfectly safe but IMO not when you are rendering for days/ weeks.

I know the Xeons are much more expensive but if you went with two 3930K you would need two power supplies, two sets of RAM, second Windows licence, maybe even a GPU.

Besides, when I think about what VRay Light Cache looks like with 32 threads I can't help but drool

Anyway, thanks for the information and enjoy your system.

[calinlihet]

The fact is I found a sort of a bargain on a German website. A Fujitsu chassis. With these 2 babies but no decent gpu and only 32gb ram for 4000 eurosI had to add in 2 ssd and a gpu... For that money I could have afford another 3930k, ram, windows and almost a second.still the sound it makes when rendering... Wow.you'd expect lightspeeds. I guess I was just over enthusiastic.

[Dimitris Tolios]

Hi Calin,

I am not sure why would you expect anything more than twice the speed of the 3930K. In fact I think you should feel lucky you are getting that much. I know one can overclock their 3930K and reduce the difference but I never liked the idea of overclocking of a workstation. It's easy and sometimes perfectly safe but IMO not when you are rendering for days/ weeks.

I know the Xeons are much more expensive but if you went with two 3930K you would need two power supplies, two sets of RAM, second Windows licence, maybe even a GPU.

Besides, when I think about what VRay Light Cache looks like with 32 threads I can't help but drool

Anyway, thanks for the information and enjoy your system.

 

High end Xeon parts are expensive - the E5-2687W particularly is $1800+ a piece...

You can almost build 3* complete 3930K/4930K nodes with the money you need just for the 2*CPUs.

 

And it is not like you don't need twice the stuff for a 2P s2011 system...the motherboard is almost 2* as expensive as two X79 boards, you need more RAM per box (almost the same per CPU) for it to have enough to feed all those threads etc etc.

 

It is space and clutter/time setting up savings more than anything with the multi-CPU systems, not money really.

Money could be of an issue ofc with render spawner licenses etc: think Vray 3.0 dropped to 3 nodes from 10 per license? even if I'm wrong, with 2P systems you could throw twice as many CPUs in your farm, and although VRay might not be the most expensive piece of software to acquire additional licenses for, could be the case that along with OS and this and that it would add up).

 

But 2P is 2P...call it a fetish... =)

[branskyj]

Hmmm, I am not normally so silly, honestly

I should have realized what a stupid thing I said there but I just came back from work, was hungry and was munching on a delicious sandwich

All true but if I had the money (and chances are I soon will) I would go for a 2P system and maybe add one more 2700K as a render slave.

As for that deal you got Calin- kudos man. I wish I have that luck here in UK soon.

[branskyj]

If you don't know the answers to this, maybe building server-based workstation isn't such a good idea ;- ).

 

With that, Quad-socket xeons aren't currently out yet at all and won't be atleast for few months I think.

Nonetheless, they won't be affordable. Keyword here is space effieciency, not raw power.

 

http://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors#Xeon_E5-4xxx_v2_.28quad-processor.29

 

Dual sockets are available, and decently good deal.

 

http://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors#Xeon_E5-2xxx_v2_.28dual-processor.29

 

Because we want them to work as powerful workstation/render station alike, it's really important to buy the correct ones, ie. the highly clocked 8 and 10 cores, not the low-clocked versions, even if they have higher core count (12). Some people have this weird idea that more cores is always better, and then they buy 2Ghz clocked weaklings ! I would laught, but these are some otherwise knowledgeable 3D folks...

 

The afforable workstation boards lg2011 are from Asus and Micro alike, with the Asus probably being rather better choice, although slightly more expensive. E5 26xx v2 are supported with UEFI(Bios) update.

 

You can buy a single Xeon for start, but take in mind the memory modules are allocated per CPU, and even then, it might be hassle. It's not advised.

 

I think decent price point where dual-socket workstation based on e5 26xx v2 start to make sense is around 6000+ euros (at this price, you roughly get performance of clocked pair of 4930k nodes). So it's not the price you save, but you end up with single powerful workstation, which I think is always better choice and the price difference is eventually negligible in grand scheme. Of course, those who enjoy servising 10 little boxes around...everyone's choice.

 

Maybe if you want to see how such workstation would look like, go to BOXX's website, look for some above 40 000 euros computer, than divide that number by 6, and you can build identical setup = ) Just copy the parts (apart from ugly Case, that one is custom)

 

Hi Juraj. I recently stumbled upon some of your renders- epic. Remarkable work, mate.

Our company too recently bought Marvellous Designer and it is quite a handy tool.

I' ll have an ale for you now .

Have a good weekend.

[Dimitris Tolios]

Hmmm, I am not normally so silly, honestly

I should have realized what a stupid thing I said there but I just came back from work, was hungry and was munching on a delicious sandwich

All true but if I had the money (and chances are I soon will) I would go for a 2P system and maybe add one more 2700K as a render slave.

As for that deal you got Calin- kudos man. I wish I have that luck here in UK soon.

 

Tell me about silly things later today...(here in the pacific, its still morning) =)

 

A 2P with that fast CPUs is great, but for most ppl in our field, a Haswell 4770K (or 4771 if you don't want the overclockable version, but want to squeeze those 100MHz over the 4770) will be faster than any Ivy Bridge Xeon or i7 (49xx).

 

For most it is counter-intuitive, but actually the 2P is better of as the dedicated render "slave" of the i7, than the other way around.

[calinlihet]

That's exactly what I plan to tr. This weekend. Put my i7 as server and the 2p just as slave. See if it speeds up my render times. This is a cool thread. Glad I stumbled upon it. And as for the mini tower I found it was on some site called gekko computer or something. I'm from romania. They shipped it using ups. I think they're refurbished. For example my chassis came with a hdd and no other holders for it (the Fujitsu has some fancy 8 slot sas în the front but no holders for more than one hard drive ) so I'm guessing it's been used before. Still 2 processors and 7 coolers inside is a geek dream come true. I called mine "the beast"

[paneli]

Julian, if you check that wikipedia page, you will see the 4-socket CPUs will not be available in powerful clocked version like the 2697 E2 and they might possibly be more than 50perc. expensive per unit as well. I would really forget about the 4unit node. There is also big chance that no E-ATX (for example the Micro is even regular ATX and the Asus is E-ATX sized SSI CEB board so with some "handy work" it can be fitted into regular XL Case) will be made available for this platform since under no case is it by anyone considered workstation material. And I would highly suggest not building a 4U rack case.

 

I though, completely agree with the logic of buying 2-socket top-range xeons. Managing 10-nodes, while cost effective, is pure craziness... I hate these budget Ikea solutions.

 

Well of course you get problems when you store all-things locally, I think investing in good NAS is right now the best investment to do. I use Synology with link aggregation and I get 160MB! transfer rates through switch with writting speeds, and this is using regular Western Red(7200rpm) disks.

 

I value comfort much more than price effectivness. My nerves and time are more valuable. But dreaming of 4-socket xeons on currect architecture is building castles in the sky :- )

 

 

 

I'll tell you this summer :- ) I plan on ordering some refurbished Dell switch and the cards are ok price. In meantime I set-up link aggregation (Asus x-79 deluxe has double 1gbit ports) through workstation to NAS and man..even without SSD it's just nice. But once going 10gbit and SSDs in file server, I can't even image...that will be heaven.

 

 

About that.

What is your transfer rate without Link Aggregation? I reach 102MB/s top with DS412+ (with the same WD Red)

One of my worksstations is also the P9X79 Deluxe but the second one is X79 Sabertooth.

 

What I mean is that I have to get a dual 1gb port on both ends? on both stations?

 

 

And also, what kind a switch did use use for that.

My local dealer has no clue about such thing...

[RyderSK]

edit: duh, I wrote bullshit due to not paying attention to your question correctly. You state without LA which is just like yours.

I guess I'll just better describe my whole setup and do some benchmarks to give it more clarity without make making stupid claims and assumptions.

Edited March 23, 2014 by RyderSK

[RyderSK]

Part 2:

I started building my nodes as dual E5-2680 v2, first two should be completed by Friday. I'll post some picks and benchmarks, how they compare to my regular 4930k. I had to already go with xeons because I already have 5x4930k as workstations/ workstation-nodes and I simply do not with to clutter my room with any more boxes to manage. Comfort of silent lower-clocked xeons seemed what I need now. So yay, my first 2p systems :- )

[paneli]

Could you direct me to the specific model of the switch?

I couldn't find it.