40 Node cooperative rendering test

[Devin Johnston]

Ok so last night I had to render out one of my projects using Maxwell and cooperative rendering. I used 40 computers of different speed and let them render for 13 hours, none of them reached a sample level that was over 10. The original images were 3000x2550 and I had to merge all 40 images together manually, these are the results.

You can see that the biggest reduction of noise occurs between 1 & 10 nodes, after that for each additional 10 nodes the noise seems to only be reduced half as much as the time before. My conclusion is that even with 80 machines rendering all night long it is still imposable to remove all of the noise. The 40 PC's that rendered this image all had 2 Gigs of ram, and any machines that tried to render it with less than that either crashed or simply never finished.

Cooperative rendering is working although it is extremely difficult to get it up and running and producing. My IT guy has had to write a program that basically copies and renames all of the MXI files from their root folder into a network directory, while it deletes all of the MXI and MXS files. We have to run Maxwell as a service on all the render nodes and we have also had to write a service program for this to work since Maxwell can't be run as a service as it is. A third program had to be written so that the service on each machine could be shutdown and restarted since for some reason Maxwell seems to remember only the last scene it rendered and no new scenes can be added. My conclusion is that although it is working it is seriously paralyzed in many key areas which need immediate attention.

[Fran]

Good demonstration Devin. So Image "1" is the 13 hour render from just one node and "10" is the 13 hour renders from 10 nodes combined, etc?

[Devin Johnston]

Yes that's it! Also I ran the same image over the weekend on my dual core dual processor machine for 63 hours and the result was virtually the same as the 40 node image.

[CHE]

Nice test Devin. I read your post in the Maxwell forum and saw the close-ups . I was hoping that with 40 computers rendering for 13hrs each there would be zero noise in the rendering.

How many plastic materials do you have in the scene? Have you try using diffuse materials for some of them instead?

Thanks for sharing.

[AdamT]

So much for NL's theory that cooperative rendering marks the end of speed concerns.

[Devin Johnston]

So far I'm under whelmed with everything about V1, for starters the fact it's being called V1 and not RC1 needs to be corrected.

[CHE]

The fact that you had to ask your IT guy to write some programs to get Maxwell to work in cooperative mode is just terrible. NL should have tested and corrected all of these issues a while back.

 

The following is just hilarious now that I read it again:

 

 

Received on 02/27/06

 

Dear customer,

 

We are writing to you to share some new and exciting developments on the road to Maxwell Render v1.0. We have set up some comparisons we have done in our testing laboratory to check the capabilities of the new cooperative rendering feature.

 

The process followed by the cooperative rendering feature is to make all the machines on the same network work on the same frame at a low quality level. Then, after the specified render time, the cooperative rendering system unites all the renders from the different machines and these results in a superior quality image.

 

This means that with the new cooperative rendering, time barriers are broken and are no longer an issue. The more machines you use, the faster and the more accurate the results.

 

We hope you find it interesting and if you have any query regarding this, please let us know.

 

 

 

Best regards,

 

 

 

Next Limit Technologies

[Ernest Burden III]

Thank you for the test, Devin. Let's see--double the nodes get 1/2 the resulting quality increase. Sounds like they've implemented inverse-square falloff, pretty physically acurate, what's the problem?

 

Makes you want to run out and buy another 80 machines, doesn't it?

[Devin Johnston]

That statement is partially true, cooperative rendering is faster than using one computer. What they leave out is you can't run cooperative rendering as a service so you have to come up with a way to trick your system into running it. Once it's running and working on a job you’re unable to add or delete servers from the job you've started. If any nodes drop off during the process the merge function will fail, that's if the job stops when it was supposed to which only happens 50% of the time. From that point it's a process of collecting all the MXI's, renaming them and merging them to get to the final product which could take hours if you did it all manually. All of this is hinging on if the Maxwell manager doesn’t crash as some point during the night. It's truly a nightmare to work with.

[Ernest Burden III]

From that point it's a process of collecting all the MXI's, renaming them and merging them to get to the final product which could take hours if you did it all manually...

 

How is it supposed to work?

[Bongo51]

remember the pretty V1 'shpeel' on their website showing co-op rendering as this child-like simple tool... press a button and it will work.

 

your telling us that you can't use Maxwell like any other network rendering tool? Like Shade, which uses other computer's 'cpus' to assist in rendering a single image? You actually have to use 20 copies of maxwell to render, then merge them all in photoshop? That's not actually network rendering... that's using 20 computers to render the same thing, then merging them in photoshop... couldn't you do that with the alpha?

 

The word cooperative...

-Done in cooperation with others: a cooperative effort.

-Marked by willingness to cooperate; compliant: a cooperative patient.

-Of, relating to, or formed as an enterprise or organization jointly owned or managed by those who use its facilities or services: a cooperative department store; cooperative apartment buildings.

 

If you need to install maxwell on 20 machines... render them all independently, collect and compile them into a single image yourself with photoshop... that's anything but cooperative.

 

more false advertising it seems.

[Devin Johnston]

Ok the way it works is it renders out the entire scene on each machine, this will result in each scene receiving a different sample level. The MXI files are supposed to be automatically merged together once this process is done but there is a bug somewhere that keeps the merged files from being saved to a network folder and it gets lost. The MXI files contain all lighting data for the scene and since there are 20 of them and the noise patterns are random each additional MXI added will result in a less noisy final image. Your basically adding light data for every extra MXI added, this isn't the same thing as combining 20 images in PhotoShop, doing that would only give you the average of all the scenes. Cooperative rendering allows you to reach higher sample levels which results in less noise and more accurate images.

 

The cooperative interface is relatively simple but it has many problems, BackBurner is a God compared to Maxwell network rendering.

[Bongo51]

Ahhh, ok. So essentially one more thing which someday will work quite well, but for V1 is only half baked..

 

hehe half baked.. I kill myself.

[renato]

anyway, the images look very CGish. They don't have the "photoreal" feeling of the beta.

[jackb602]

Thanks for the test Maxer. I looked at the renderings more closely and I think the 30 node rendering is the one that really hits the sweet spot. The sunlight is more pronounced and realistic in that version I think. It's cool that you've gotten it to work, but depressing that you need 30 nodes and an IT department to generate a single Maxwell render. Looks like I'll be sticking with Cinema's Advanced Render for now.

 

Jack

[Ernest Burden III]

Looks like I'll be sticking with Cinema's Advanced Render for now.

 

Oh, don't do that. Haven't you been reading cgtalk lately?

[jackb602]

Hey Ernest. Yes, I've seen your painful AR experience documented at CGtalk; sorry you had to go through such a nightmare. I've dealt with similar crisis situations myself, but on a smaller scale. Advanced Render is by no means perfect, but unless I leave C4D it's the only game in town for me at the moment. finalRender looks promising, but still seems to have some substantial bugs and of course won't be out for the Mac for several months. Vray sounds like it might be a great option in the future as well, but I suspect that is several months off as well.

 

I'm also optimistic that Maxon will have something new for us at Siggraph. After all they've just released MoGraph, and I don't think they're going to show up there empty handed. How did your project turn out in the end?

 

Jack

[AdamT]

True, AR has problems with GI animation, but it's fine for stills. No bucket rendering though, which is a drag. I'm looking forward to VRay4C4D.

[Ernest Burden III]

True, AR has problems with GI animation, but it's fine for stills. No bucket rendering though, which is a drag. I'm looking forward to VRay4C4D.

 

AR is not fine for stills, except if you have a really long time to wait for them. It produces circular artifacts with anything but really high settings (again--long times involved) etc. I know its easier to wait a while longer for one still, but after a year I am not finding AR to be adequate for architectural GI work. It doesn't suck, but it also has left me disappointed and frustrated on every project I've used it on. It's need an overhaul badly.

 

And if you're on Mac I see the point of how the options are thinner, so you should stick with it for the time being.

[AdamT]

Well, as we've learned with Maxwell, slow is a relative term (like "soon"). I can do print resolution interiors with AR on my 3-comp network overnight that would take a week with Maxwell.

 

fR is definitely faster with GI, but AR is faster at raytracing.

[Ernest Burden III]

I can do print resolution interiors with AR on my 3-comp network overnight that would take a week with Maxwell.

 

Networked computing certainly works with C4D, but on one image--are you splitting up the picture into tiles, because Cinema's engine doesn't break up individual images (which you know but others reading this might not)?

[AdamT]

Networked computing certainly works with C4D, but on one image--are you splitting up the picture into tiles, because Cinema's engine doesn't break up individual images (which you know but others reading this might not)?

Yes, using the preset tiled camera you can render out x number of image strips, which you then assemble in PS. Another issue is that it can affect GI sample distribution, but that can be dealt with in PS too.

[AdamT]

[delete duplicate post]

[Devin Johnston]

I did a little more testing last night; I think I had about 70 or 80 machines online. One problem I've discovered is that the Manager crash during the merge function on large jobs, I don't know what is going on there. Another very annoying bug or lack of functionality is that the manager doesn’t remember any job sent to it once it's been shut down, so it's imposable to restart anything unless you want to completely start over. It also forgets any network rendering groups you create so there only good for as long as the manager us up and running. I'm going to start the process of pulling all the MXI's together and I'll post something later today.

[Adehus]

I'm going to start the process of pulling all the MXI's together and I'll post something later today.

 

I'll be curious to see what you come up with. While I can only dream of having as much computing power as you have(!), I appreciate you sharing your findings with us.