Rendering Cluster Super Computer

[itsallgonewronghere]

Does anyone have any experience of using something like a Beowulf cluster or Windows HPC for rendering (specifically Max+Vray)?

 

We have a large team with something like 300 workstations. Most are quad cores with 8GB of Ram, but most of these are being used by users / applications that only ever utilise 1 or 2 cores at any one time.

What I would like to do is take say half those resources from every workstation and network them into 1 cluster supercomputer.

So taking:

- 2 cores per workstation x 300 workstations = 600 cores!!

- 3GB ram per workstation x 300 = 900GB ram!!!

So when we render out large stills instead of waiting 8 hours for a single machine or rendering in strips (less time consuming but messier) we can have the "supercomputer" open 1 instance of the scene and use all 600 cores to render the still (in minutes)!

 

Is this possible?

Anyone tried anything similair?

Any advise?

[Frosty]

Sort of. Read up on Back Burner.

[MD]

Would you need to have max + vray installed on each machine? I'm interested to hear what you find out.

[Slinger]

Chaos Group products are licensed per machine and with one V-Ray for 3ds Max license you can start and use V-Ray GUI and tools inside one 3ds Max platform, start one local render job, start Network rendering with unlimited render slaves using “3dsmaxcmd.exe” or job manager like Backburner, start up to 10 slaves for Distributed rendering using V-Ray DR spawner.

[neil poppleton]

Wow Distributed rendering and steal all the power. max 10 pcs....still a lot of power. And slaves of 300 for your animations....nice..

[itsallgonewronghere]

What I am trying to get away from is distribution rendering and Backburner. For various reasons including:

- with very very large scenes sent to backburner the files need to opened in max on every machine it gets sent to by backburner. this can be very time consuming (if the scene takes 20 minutes to load everytime), result in machines crashing/freezing.

- all plugins need to be installed on all the machines

- if you send a large still render to backburner as strips, the results are never 100% right, slight differences in lighting (even with precalculated GI)

- rendering in strips will give you a stitched together image but does not seem to be able to do the same for the render elements (reflection, refraction, lighting ....)

 

With the scenario I am hoping to setup instead of sending a job to a network of computers which has every machine assigned its own job to do (ie backburner), mine would have all the machines behaving as 1 machine (ie opens 1 session of max). The whole network would basically behave like 1 computer with 300 cores!

[Scott Schroeder]

My advice is stay away from backburner for this type of operation. Backburner is not recommended for anything more than 10 or so machines. Yes, it will work on this large, but trust me it will give you headaches. What you are doing is still distributed rendering. You just have one machine talking to all of the other ones.

 

The issue you will see is that even though the user computer is only taking one core, when Vray gets going on that machine it'll still slow things way down. Rendering is a beast, even if you separate the cores. And sending out that kind of network traffic, I hope you've got a solid pipeline and that the rest of the company is only transferring 14k word docs. Remember, even if the user is using 2 of the 4 cores and now you are using the other 2. You've now got all 4 cores spiked at full use. It will slow the user down.

 

Besides, 300 buckets? You'd crash you own system before you got anything done. What's Vray's limit on buckets? Unless you are rendering at 400,000 pixels, 300 cores on only one machine is such overkill. You'd be better off at 150 separate machines. Double up your cores on single machines, and have this vast render farm.

 

If you want to make it a true clustered super computer where you can actually share the RAM, you'd need to really read up on how to do that (ie go classes for IT) or hire someone who has this type of experience.

 

We have a similar setup at Purdue University on our Coates Super Computer and it only took 2 years of R&D to actually get Vray to work correctly. http://en.wikipedia.org/wiki/Coates_%28supercomputer%29 We had use a custom written Condor based distributed rendering solution. Realistically, we only use it to render 4k video. For your average daily stills, this thing is just way too strong.

[kmanus]

Someone correct me if I'm wrong since I use MR and not vRay (although I don't think it matters in this case), but aren't you limited to 8 physical cpu's when doing a DBR render? Wouldn't that make what he is proposing impossible? Seems like your best bet would be to set up one workstation and 7 slaves if you insist on rendering in the fashion you've laid out here, although DBR renders have their own set of problems...

[Scott Schroeder]

I think that's the limit of mental ray, and not Vray. MR is licensed per CPU and Vray is per computer with a high limit on network rendering. I think 1,000 machines? Vray does have a 10 machine DR limit per license. So if you have 4 licenses of Vray you can have up to 40 DR machines. MR is different, unless they have changed it recently, but it is per CPU so if you have a dual quad core machine, you need 2 MR licenses to use each CPU properly.

 

However, Vray does have an upper limit on the number of buckets it uses before it just becomes a waste of resources. Meaning that by the time the information is sent out to all of the different nodes, the first 4-5 nodes have already completed 95% of the image. The only way to solve that issue is to have the machines all wait until everyone has gotten the information and then render all at once.

[sayan]

Does anyone have any experience of using something like a Beowulf cluster or Windows HPC for rendering (specifically Max+Vray)?

 

We have a large team with something like 300 workstations. Most are quad cores with 8GB of Ram, but most of these are being used by users / applications that only ever utilise 1 or 2 cores at any one time.

What I would like to do is take say half those resources from every workstation and network them into 1 cluster supercomputer.

So taking:

- 2 cores per workstation x 300 workstations = 600 cores!!

- 3GB ram per workstation x 300 = 900GB ram!!!

So when we render out large stills instead of waiting 8 hours for a single machine or rendering in strips (less time consuming but messier) we can have the "supercomputer" open 1 instance of the scene and use all 600 cores to render the still (in minutes)!

 

Is this possible?

Anyone tried anything similair?

Any advise?

 

 

hiii...i am sayan..from india..did you make the above said rendering cluster??if yes..please give me some idea..please