New Render Nodes

[Jason Matthews]

Hey all,

 

I will be building three render nodes for 3D Max and Vray within the next month or two. Below are the specs for them. Let me know what you think and any recommendations you may have. This is a copy and paste from Newegg. I can't decide if I want to buy a MB with on-board video and save like $50 per station, or go with this rig and have the video cards for VrayRT. Can you guys recommend a video card around the $100 price range that is better for RT than the one listed?

 

COOLER MASTER Centurion 5 CAC-T05-UW Black Aluminum Bezel , SECC Chassis ATX Mid Tower Computer Case

ASUS P9X79 LE LGA 2011 Intel X79 SATA 6Gb/s USB 3.0 ATX Intel Motherboard with USB BIOS

EVGA 02G-P4-2643-KR GeForce GT 640 2GB 128-bit DDR3 PCI Express 3.0 x16 HDCP Ready Video Card

Rosewill HIVE Series HIVE-750 750W Continuous @40°C, 80 PLUS BRONZE Certified, Modular Design

Intel Core i7-3930K Sandy Bridge-E 3.2GHz (3.8GHz Turbo) LGA 2011 130W Six-Core Desktop Processor

G.SKILL Sniper Gaming Series 32GB (4 x 8GB) 240-Pin DDR3 SDRAM DDR3 1866 (PC3 14900) Desktop Memory

Western Digital Caviar Black WD1002FAEX 1TB 7200 RPM SATA 6.0Gb/s 3.5" Internal Hard Drive -Bare Drive

ASUS DRW-24B1ST/BLK/B/AS Black SATA 24X DVD Burner - Bulk - OEM

Microsoft Windows 7 Professional SP1 64-bit - OEM

Arctic Silver 5 High-Density Polysynthetic Silver Thermal Compound AS5-3.5G - OEM

ZALMAN CNPS20LQ Ultimate Liquid CPU Cooler

 

Thanks,

[edgy_k]

The 3930K is a good choice. I built 2 render nodes with this processor and pretty much the same specs are yours (except for the graphics cards : GTX 580) I'm very satisfied with their performance.

The only advice I'd give you is to go for ssd drives. It is not essential, but that will make the render nodes start quicker and join the rendering faster.

[Jason Matthews]

I would love to add GTX 580s or even GTX 570s, but my budget doesn't quite allow it. If anyone can see a way to cut costs in this rig please advise. I do want the 6-core and 32 gigs of RAM though.

 

Since this if my first network rendering setup, can anyone suggest absolute must-haves as far as the network is concerned? I planned on running the comps over an 'N' wireless router (physically plugged in though).

Edited September 5, 2012 by Saturn

[Dimitris Tolios]

Are you looking to build workstations or render nodes?

Why bother with GTX cards? Networking GPU rendering? Otherwise render nodes are usually head-less. I don't know if there are on-board GPU enabled s2011 boards tho.

 

Afaik, GPU rendering works better locally, unless you have much more than you can fit in one workstation box.

 

If those will be purely for CPU rendering, maybe you should work the idea on getting a Quad G34 system

 

4x AMD MC 6128 8 Core Processors - $100 each - eBay

 

Supermicro H8QGL-IF - $640

 

RAM - 4 Sets of 2 or 4 Sticks -

G34s are quad channel but you won't lose much if you don't. I would get 4 sets of 2x4GB Samsung 30nm DDR3 1600 1.35v. Board does not need ECC.

 

Cooling - 4xCM 212+s - $30 each

 

PSU - Again your choice, just get a quality brand, I would recommend at least 850W. 1000-1200W if you want to have the most efficiency possible.

 

HDD - W/e. A 60GB SSD could work.

 

VGA - has a built in basic unit just to set it up.

 

Case - You will be better off building a wooden box were everything just sits in...really...

You can get SWTX cases for $200 or so.

http://www.amtrade.com/ipc/4u_quad_cpu_rack.htm

 

This system should be put together for $1600 or so + case. Depends on PSU choice.

Performance wise @ stock it should be 2x or so of a single 3930K for rendering, so you can build a 3930K workstation with 1-2-3 GPUs if you are inclined to do so, and use the 4P purely for CPU production renders. Just one node to worry about setting up, one OS licence, one HDD/SSD etc.

 

CPUs can be overclocked @ around 2.4-2.5GHz with this board and cooling - again, not a must, yet remains possible and pretty effective.

[Jason Matthews]

Thanks for the feedback, Dimitris. I looked up a lot of the info you wrote. I have no experience with server boards or with AMD chips for that matter, but I am rather intrigued. Are there any other cases that will fit this monstrous board?

[Dimitris Tolios]

The SWTX (super wide?) is a Supermicro proprietary format. Dimensions: 16.48" x 13" (418 x 330 mm).

It is hard to find cases for it, as it was marketed as a server board. There are a few expensive and hard to find cases OEM or Supermicro branded other than that posted. The system is extremly easy to setup tho, as it has no cards etc, so you don't really need a case. Most of the 4P folding rigs I've seen utilizing that board were housed in either modded, or DIY cases (made out of wood even), or just standing ontop of a box/base that contained the PSU and HDD.

 

Other options for G34 4P are coming from TYAN S8812WGM3NR - newegg.

Pricier and with no O/C support, these are usually not preffered by enthusiasts that don't look for a full scale render farm, but just to maximize their investment on one node (thus overclocking if possible). Ofc it is a good board if you don't plan to O/C.

The TYAN boards are MEB form factor tho, which does fit in a selection of 4U cases/ big-Full towers etc. .

 

If I would get the SWTX board, I would make a small box in the size of the board (well, not exactly small, the board is huge) and with inner height that of the PSU. Probably out of MDF that paints well. Mount PSU / HDD / power button (DIY or you can order power buttons with leads+2pin connector for $5-10) inside the box, screw some stand-offs on the side to mount the board, and hang it on my wall.

 

Both of these boards are used as render nodes and/or protein folding rigs (Folding@Home), and I know of people that have been working them 24/7 @ 100% for a couple of years with no mishaps - even overclocked. My estimation on 2x 3930K performance are coming from these real life numbers: O/Ced 6128 @ 2.5GHz does 360-380K PPD with F@H, overclocked 3930K does around 140-150K PPD.

In Cinebench 11.5 and stock speeds it scores nearly 2x of a typical stock 3930K.

 

ebay: did check on ebay, there are 5-6 pieces of 6128 2GHz available from $80-105. Most of them from 100% rated sellers that are most likely IT pros, and are selling CPUs out of systems that get upgraded to 16cores etc. The chip new is $250+, and I would not hesitate to get it used to save that kind of money.

Edited September 6, 2012 by dtolios

[Jason Matthews]

Interesting. I was searching everywhere for a case that could house this motherboard and every site I went to basically said the same thing. I did find a case that supposedly works. The Xigmatek Elysium: http://www.newegg.com/Product/Product.aspx?Item=N82E16811815011

 

I don't know how comfortable I am building an MDF box for a motherboard. I would need to get the holes absolutely perfect for mounting.

[Dimitris Tolios]

 

I don't know how comfortable I am building an MDF box for a motherboard. I would need to get the holes absolutely perfect for mounting.

 

Meh...a 2mm mechanical pencil lead can do the trick perfectly, using the board itself as a guide. The motherboard stand-offs are coarse threaded and will bite into the MDF enough. Add some wood glue in the hole if you want more security. Countersunk screws, some wood filler and a clean paintjob can make a great "conversation piece".

 

Other DIY ideas:

http://www.overclock.net/t/1289873/build-log-4p-24-7-folder-and-file-server-spartacus/0_50#post_17906634

http://www.overclock.net/t/1264004/build-log-another-4p-now-overclocked/0_50#post_17370958

http://www.overclock.net/t/1239806/build-log-g3rgs-custom-wood-4p-case/0_50#post_16911111

 

 

One of the Elysium mods you've mentioned: http://www.overclock.net/t/1241247/4p-buildlog/0_50 (the guy has a cinebench 11.5 @ 40+...woot).

 

Used to be fond of the idea building cheaper, smaller nodes using i5s or AMD FX CPUs,

but these 4P rigs, despite requiring some tlc with the case etc, appear to be a better investment:

• 1/4-1/5th the cost for the OS
  
• 1/4-1/5th the cost for HDD/SSDs. Everything will fit in a 60-64GB SSD nice and easy. Those are dirt cheap now.
  
• one big efficient PSU that is cheaper than 4-5 cheap units you would use to match the performance with i5-like CPUs - tho with some modding (again) you can power 2x mATX boards + CPU with a 500W or so PSU.
• less things to setup software wise
• definitely less space needed - unless you manage to do one of those "ikea helmer" clusters #1 #2 #3

 

On the con's are the worse redundancy - if your single 4P node fails, you will have downtime till you mend it, while if you have more little ones, and one fails, you still run @ slightly reduced speed.

Edited September 6, 2012 by dtolios

[Cjmcphail]

Are you looking to build workstations or render nodes?

Why bother with GTX cards? Networking GPU rendering? Otherwise render nodes are usually head-less. I don't know if there are on-board GPU enabled s2011 boards tho.

 

Afaik, GPU rendering works better locally, unless you have much more than you can fit in one workstation box.

 

If those will be purely for CPU rendering, maybe you should work the idea on getting a Quad G34 system

 

4x AMD MC 6128 8 Core Processors - $100 each - eBay

 

Supermicro H8QGL-IF - $640

 

RAM - 4 Sets of 2 or 4 Sticks -

G34s are quad channel but you won't lose much if you don't. I would get 4 sets of 2x4GB Samsung 30nm DDR3 1600 1.35v. Board does not need ECC.

 

Cooling - 4xCM 212+s - $30 each

 

PSU - Again your choice, just get a quality brand, I would recommend at least 850W. 1000-1200W if you want to have the most efficiency possible.

 

HDD - W/e. A 60GB SSD could work.

 

VGA - has a built in basic unit just to set it up.

 

Case - You will be better off building a wooden box were everything just sits in...really...

You can get SWTX cases for $200 or so.

http://www.amtrade.com/ipc/4u_quad_cpu_rack.htm

 

This system should be put together for $1600 or so + case. Depends on PSU choice.

Performance wise @ stock it should be 2x or so of a single 3930K for rendering, so you can build a 3930K workstation with 1-2-3 GPUs if you are inclined to do so, and use the 4P purely for CPU production renders. Just one node to worry about setting up, one OS licence, one HDD/SSD etc.

 

CPUs can be overclocked @ around 2.4-2.5GHz with this board and cooling - again, not a must, yet remains possible and pretty effective.

 

What OS would you use for a beast like this? I am also in the market for a render node. I was going to go with, Intel Core i7 3770 a cheap motherboard with on board graphics 16 gb ram and a miniATX case with psu.

[Dimitris Tolios]

Extremely good catch...I ranted a lot without double checking on that...important requirement...

Windows 7 Pro/Ult are limited to 2x physical CPUs. Same for Win XP 64 Pro/Enter.

That means that the most "economical" Win OS with full support for 4x physical CPUs should be Windows Small Business Server 2011. The "Essentials" version does retail around $390. So much for big savings on OS licences

 

I got pwned.

 

----

 

For a small mATX node, Win 7 is more than enough, depending on your configuration.

 

For up to 16GB of RAM, Win 7 64bit Home Prem is enough.

For more than 16GB you need Pro/Enterprise/Ultimate.

 

Pro/Enterprise/Ultimate do support up to 32 logical processors, but are limited to 2x physical CPUs (2 sockets), while Home editions allow for only 1x CPU.

 

----

 

Thanks again for pointing out the obvious mistake.

Edited September 7, 2012 by dtolios

[Cjmcphail]

You help is great dude. Im just really stuck on what to build. I want to get as many Cores as I can for as little $$$, so i thought i would do my research etc

[Dimitris Tolios]

AMD 8120 is not a bad choice for a render node. Slightly faster than the i5 2500/3570, and it is noticeably cheaper than a i7 if you are after a budget system.

 

In general CGarchitect does not encourage overclocking, and it is true that for inexperienced users it might cause more problems than it solves. But tbh, it is so easy and effective in the latest generation of Intel i5/i7 K series and the AMD FX bulldozer chips, that it is a shame not to try. The 8120 is considerably cheaper than the 8150, while being essentially the exact same chip. Both are unlocked - unlike intel where only the top-tier i5 & i7 is unlocked (K version), and both can hit 4.4-4.5GHz which offers huge rendering performance increase - especially for the 8120 which defaults @ 3.1GHz.

 

Fully fledged mATX boards for FX 8 cores are harder to come by than with LGA 1155, but there are some good ones available nontheless.

 

ASUS M5A88-M AM3+ is a nice choice - not the cheapest around @ $100, but is an OK overclocker and has 4x Dimm slots - important if you want to use smaller dimms to get to 16GB of RAM - ceiling for this board. Added bonus: USB 3.0 support (not native to the chipset).

 

GIGABYTE GA-880GM-D2H

This is another decent board. Cheaper, @ $75 - has 2x Dimms and it is limited to 8GB of RAM...otherwise pretty much all the features the ASUS has, but no USB 3.0 - I don't know how important this is for a render node, but had to find something. Surely the 8GB cap hurts more - but you can work with it for simpler stuff or by rendering @ raw files directly in a HDD location instead of your Vray buffer = RAM.

 

Unfortunately the 9xx AMD chipsets are not represented in the mATX form factor. If you could do ATX (you weren't after a Helmer or other type of SFF cluster) I would definitely do it.

 

RAM wise, I cannot stretch enough how satisfied I am using the Samsung 30nm, low profile PC1600 1.35V 4GB sticks. Unfortunately that's the biggest those chips get on a single dimm, but don't be fooled: those are some of the best RAM chips you can buy. Rated @ both 1.35V and 1.5V, those can hit 1866 with very low timings @ 1.35, and I run my 2 pairs @ 1.5V and 2400 (1200MHz DDR) 11-11-11 1T...no tweaking, no nothing. It just works. It is amazing if you think that most 1866 kits are rated @ 1.5-1.6V and almost all 2133 and faster kits are 1.65V, when these sticks with no fancy heatsinks, do much much more for $45-50 a pair.

 

Sorry for linking US stores - that's what I find handy. Most of these items should be available worldwide, yet pricing will most likely be higher.

 

AMD chips are very memory bandwidth dependent, and would appreciate the added speed much more than my i7 3820 does. I compromised with getting 4x4GB while I was going for 4x8GB, and I hope that since I have 8 dimm-slots total, I will upgrade to 8x4GB while maintaining my O/C.

Edited September 7, 2012 by dtolios

[Cjmcphail]

Have you heard of or know anyone using the server board setup you have mentioned in previous post? I like the idea of having one box with 4 CPUs

in it

[Dimitris Tolios]

Have you heard of or know anyone using the server board setup you have mentioned in previous post? I like the idea of having one box with 4 CPUs

in it

 

I do, but they are running Linux and using it for Folding, not rendering, so the question around windows never came up.

 

There are rumors that Win 8 will support 4 CPUs when fully released (the current beta does), and ofc the code to do it is there, but just like with Win7, microsoft might end up removing the support for more than 2x CPUs to promote the Win Server (non-Home) versions.

Edited September 7, 2012 by dtolios

[Cjmcphail]

Maybe multiple mother boards in a custom box is the way to go

[Dimitris Tolios]

Maybe multiple mother boards in a custom box is the way to go

If you plan on using multiple nodes and you don't care much for the commercialized approach (expensive rack mounts), then DIY boxes or modular trays are much more popular than you think - even for university super-computer clusters.

 

Personally I find these DIY cases very appealing, compact and effective

 

bunnyburg, by brave rabbit -

Another - very clean - helmer "hackintosh" cluster, using i7s + ITX boards, though it has poor thermals due to the lack of cross-flow (no forced intakes - he tried too hard to preserve the front of the drawer).

Edited September 7, 2012 by dtolios

[Smile of Fury]

It looks like you are in a way higher price bracket, but I just built two inexpensive nodes based on FX-8120. Thought I'd share the build here. I'm an arch. grad student, so these nodes aren't really generating any income for me. Tried to keep the price down as much as possible, but ended up getting a larger case and nicer mobo so I have room to expand these into workstations if I need to, or maybe sell to some other students.

 

With mail-in rebates and other deals the final cost was about $425 a node. That was before the OS. The links are all NewEgg, but I got half the stuff at MicroCenter. For the total price I could have probably made one 3930K build, but I'm pretty happy with the results.

 

CPU: FX-8120

MOBO: ASUS M5A97 (The CPU and MB came packaged at MicroCenter for $190. I wanted the M5A88 Dmitris just mentioned, but they didn't have it. Ended up having to throw in some $15 video cards because I couldn't get the 97 to boot without a GPU. My nodes are not connected to monitors, so I got the cheapest GPU I could find since they do nothing but allow the computer to boot.)

RAM: Corsair Vengeance 1600 (4x4GB)

SSD: OCZ Agility 3 SSD

COOLING: Zalman CNSP 9500A

CASE: Cooler Master Elite 371

PSU: Corsair CX430

 

My workstation is based on i7-2600. Each node is slower than that, but together they render faster than my workstation. A small scene 300 frame animation took about 45 minutes with all three comps working on it. Would have been about 2hrs on just my workstation.

 

Oh yeah, and thanks Dmitris. Could not have done it without all the info you provide here every day.

[Cjmcphail]

It looks like you are in a way higher price bracket, but I just built two inexpensive nodes based on FX-8120. Thought I'd share the build here. I'm an arch. grad student, so these nodes aren't really generating any income for me. Tried to keep the price down as much as possible, but ended up getting a larger case and nicer mobo so I have room to expand these into workstations if I need to, or maybe sell to some other students.

 

With mail-in rebates and other deals the final cost was about $425 a node. That was before the OS. The links are all NewEgg, but I got half the stuff at MicroCenter. For the total price I could have probably made one 3930K build, but I'm pretty happy with the results.

 

CPU: FX-8120

MOBO: ASUS M5A97 (The CPU and MB came packaged at MicroCenter for $190. I wanted the M5A88 Dmitris just mentioned, but they didn't have it. Ended up having to throw in some $15 video cards because I couldn't get the 97 to boot without a GPU. My nodes are not connected to monitors, so I got the cheapest GPU I could find since they do nothing but allow the computer to boot.)

RAM: Corsair Vengeance 1600 (4x4GB)

SSD: OCZ Agility 3 SSD

COOLING: Zalman CNSP 9500A

CASE: Cooler Master Elite 371

PSU: Corsair CX430

 

My workstation is based on i7-2600. Each node is slower than that, but together they render faster than my workstation. A small scene 300 frame animation took about 45 minutes with all three comps working on it. Would have been about 2hrs on just my workstation.

 

Oh yeah, and thanks Dmitris. Could not have done it without all the info you provide here every day.

 

I'm not sure if I'm reading right but are you saying that 2 of your AMD FX8120 are rendering at a slightly faster or much faster speed as your i72600?

[Smile of Fury]

Just the i7-2600 each frame was about 20-24s. With the i7-2600 and 2x FX-8120, each frame was about 9-12s. This was with VRay DR. Had some issues with Backburner and haven't had time to investigate. I think with DR only the workstation calculates the GI and then the nodes come in just for the final pass. Is that right?

 

I'm using Max 2012 and Vray 2.20.

[Cjmcphail]

Using Vray DR My current nodes help with the IR map as I Calculate my light maps before before I render the final pass . So your nodes should help with the IR calcs. That's assuming you are using IR. I have the budget to buy 2 i7 3770 each with 16 gb ram so I might just looking into some blue prints for a customs case/rack mount

[Dimitris Tolios]

Well, should have played more with linux when I was back in high school...now the dog is getting old for new tricks...just using live USBs for recovering files is not good enough i guess.

 

Apparently you can do V-Ray DR Spawner with Linux OS + WINE, so if a render node is all you need, you could bypass the expense for MS server OS etc...ofc same goes for 1P nodes as-well.

 

If I will find a tutorial/guide I will link it...(hopefully someone will beat me to it).

[Cjmcphail]

how does that work. I didnt think Autodeks Products work on linux

[Dimitris Tolios]

Well, should have played more with linux when I was back in high school...now the dog is getting old for new tricks...just using live USBs for recovering files is not good enough i guess.

 

Apparently you can do V-Ray DR Spawner with Linux OS + WINE, so if a render node is all you need, you could bypass the expense for MS server OS etc...ofc same goes for 1P nodes as-well.

 

If I will find a tutorial/guide I will link it...(hopefully someone will beat me to it).

 

Still no luck finding a way to do it...

I think I was miss-lead by the equivalent setups with VRay for Maya, and not 3DS...

 

I see a few Render Farms that claim have their nodes running Linux 64bit and rendering 3DS fills through Vray...

Maybe a representative of such a company could share some knowledge???

 

Unless Win 8 comes out with 4 CPU support (which I doubt), I guess the 4P Opterons are limited to Win Server editions???

There are still dual socket motherboards for those - not 1/2 the price yet close, yet not the same- takes away the convenience of having only one node to setup, plus adds costs with 2x OS licences (still cheaper than a single server edition), 2x HDD, 2x PSUs etc...at least those are in form factors not that huge as the SWTX, and fit in many U-rack / full towers out there.

[Cjmcphail]

Here are some interesting builds I came across that are pretty inexpensive

 

http://digitaltimmay.com/2011/03/16/how-to-build-an-inexpensive-render-farm/

 

http://blog.dotlot.com/building-an-ikea-helmer-render-farm/

[Jason Matthews]

Well I bit the bullet today and bought two of the three stations. I substituted a 128GB Crucial SSD for the HDD for about the same price. I can always add an additional HDD to the machines if need be, but my workstation has the capacity to store all the renderings for the animations. I also added a gigabit switch and KVM switch for better functionality and connection. Now at least I won't have a bunch of keyboards and mice running around. I may in the near future by a i7-2700k system as well, but for now this will do.

 

In the end, I did not want to run Linux for the AMD server. I am very hopeful that Windows 8 will ultimately have support for 4 physical processors.

 

Thanks for all the conversations.