Loading proxy information on render.

[Crazy Homeless Guy]

I have spent my spare time during the last couple of day's setting up a test to determine the feasibility of using Mental Ray proxies in a production environment. I want to be sure that Mental Ray can handle some of the larger projects I am working on. I had planned on posting that test by now, but it became apparent that I needed to hold off on posting the initial results until I figure out exactly how the Mental Ray proxies work. I am running into a few snags. So, her is my basic problem...

 

I have a proxy tree that I created. The proxy tree has 444,100 polygons. This is more poly's than my trees typically have with the exception of maybe a every green. I placed this proxy into my Mental Ray scene, and created 300 instances of it in the scene. I hit render, and it starts processing, but it doesn't make it to the Final Gather. I am lucky to be working on a robust machine. I let it run up to 6.5 gigs of RAM before I force kill the process.

 

As I look closer at the problem, it seems like Mental Ray is trying to load the geometry for each proxy into RAM before it starts to render. This is not what I expected to happen. The disk space for this proxy is 67 megs. If Max were to load 67megs into RAM 300 times, I would be looking at around 19.6 gigs of RAM for just that. Didn't someone post a thread the other day about wanting 32 gigs of RAM? ...I think it might be useful in this case.

 

So... Not knowing if that is exactly what is going on or not, or if I am doing something wrong, I decide to do another test. I took the same proxy, and instanced it 50 times in my scene, and hit render. If my above statement is true, the the RAM should hit about 3.27 gigs plus the whatever amount is needed to load 3dsmax into memory. So, I hit render. The calculation was fairly accurate. The image started rendering when the RAM level was around 3.3 to 3.4 gigs. It may have taken slightly more RAM, the numbers were jumping around, and I wasn't keeping a close enough eye on them.

 

 

"...These bytes can then be read in... but not until a ray actually touches the bounding box!"

 

".... Not until the object is actually needed for render is it even loaded into memory, and when it is no longer needed, it can be unloaded again to make room for other data."

 

 

Obviously the proxy is being loaded into RAM, but my scene seems to be loading all of the proxies into RAM before it starts to render. Maybe I don't understand enough about how the final gather works. I assumed it would be loading the proxy for the bucket it was on, and unloading the proxy geometry after it finished the bucket. ...and even then, I would think that if it was an instanced proxy, then the geometry would only need loaded once, instead of 50 times.

 

The other thing that is really weird. My file is 34 megs with the proxies in it. This is another big flag that says I am not doing something correctly.

 

I am no Mental Ray professional, but I can't help to think that I am doing something wrong here. I am posting my scene for other to look at. I am not in the office today, so I can't run more tests. At least more tests with the original scene I have set up. I may make a simplified scene and do some more tests.

 

SCENE : http://www.phase22.com/misc/cgarchitect/mental_ray_proxies/base.zip

 

.

[Crazy Homeless Guy]

Hmmm... I am going to have to check the log files when I get to the office tomorrow. This guy seems to be pushing a 100,000 poly proxy object lots of times. He is managing 55+ million poly's in under 5 minutes at 2000x2000 pixels.

 

http://forums.cgsociety.org/showpost.php?p=5084793&postcount=529

 

.

[demo38]

That's a lot of polys. I'm curious to see what you find out.

[Macer]

Yeah, I too am interested to see how this turns out. I haven't had an opportunity to play with proxys yet. Please keep us informed....

[Crazy Homeless Guy]

The other thing that is really weird. My file is 34 megs with the proxies in it. This is another big flag that says I am not doing something correctly.

 

Well, this pas part was due to not double checking to make sure the file was properly cleaned up. There was a left over mesh that was hidden. After deleting it, the file was a more predictable size of 80k.

[Crazy Homeless Guy]

Well, I am beginning to get the scene to render. I have not got it as fast as what I would like, but I am still in the process of learning, so I won’t be overly critical of that right now. I am getting times of 24 minutes at 3000x1500 pixels with the geometry listed above. My sampling/AA is whack, so the images don’t look decent enough for production, but this is something I will deal with later. There is potential that my rendering time will take a big hit when corrected to a useable sampling/AA.

 

The main thing that is still proving problematic is the RAM consumption. I am still lingering around 3gigs needed to render this scene. My personal feeling is that this is high, and should be lower. I don’t feel the geometry is being loaded and unloaded in an optimal way. I don’t know if it has to do with the ray distribution in the scene keeping the geometry in the RAM, and not releasing it. Or if it is something I have set up wrong.

 

I re-packed the scene without the extra geometry. If someone else wants to play around with this, feel free. If you limit the test to 10 or 20 trees, the RAM consumption should be low enough that anyone can play.

 

SCENE : http://www.phase22.com/misc/cgarchitect/mental_ray_proxies/base.zip

 

.

[Crazy Homeless Guy]

If I am reading the Mental Ray Message Log correctly, it looks as if it is seeing my trees as being instanced. If instance is reffered to as Shared.

 

SCEN 0.3 info : 51 geometry leaf instances (2 scheduled, 0 cached, 49 shared)

[Crazy Homeless Guy]

Some of you may say think that I am expecting to much. I am basing my assumptions on how much RAM the scene should use from working with proxies with Vray. To put it simple, I have this scene also setup with the VRay proxies. It renders the entire scene using less than 650 megs of RAM. This is why I am assuming that I have something out of whack.

[pixelperfectg]

Switch over to BSP2. I'm not sure why many people are overlooking this, perhaps it wasn't focused on enough in the documentation? BSP2 is much more efficient at handling millions of instances.

 

Using my 3-4 year old 32-bit system with 2gb ram I can easily render over 600 instanced copies of your tree mesh.

 

(I stopped the render because I have work to do and couldn't tie up my PC for the full render).

 

Bottom line, when using a lot of instanced proxy objects, you should be using BSP2 for optimal results.

[Crazy Homeless Guy]

That totally kicked ass, and was the exact setting I was looking for. Thanks.

 

It used under 1 gig, and render in 1 minute and 34 seconds with my current settings.

[dagon1978]

jeff allready give you the right direction

i can add a little thing, if you want to push a little bit more your system you can try to disable the "scanline" algorithm (using the raytracing for primary rays)

 

here's a test with your scene and 1000 instance... but i have a 64bit system with 8gb RAM

 

 

ciau

 

mat

[pixelperfectg]

but i have a 64bit system with 8gb RAM

Sure go ahead and rub my nose in it why don't ya!..lol (just kidding with you). I noticed your signature, congrats on the mr book!

[MasterZap]

Switch over to BSP2.

 

Not only that - switch to BSP2 *and* turn of "Scanline". Scanline is neat and speeds up simple scenes. It doesn't really speed up *complex* scenes that much, and, also, creates some memory consumption that isn't flushable.

 

Lemme know the memory consumption after doing that...

 

/Z

[dagon1978]

Sure go ahead and rub my nose in it why don't ya!..lol (just kidding with you).

eheh it's my new baby... i'm waiting to replace my two dual with two quad now

 

I noticed your signature, congrats on the mr book!

 

thanx jeff... we start to print next week and we are planning the english translation right now...

oh, i added your site and blog on the links page, are very insipring for all the mray community... i hope is not a problem for you

[dagon1978]

Not only that - switch to BSP2 *and* turn of "Scanline". Scanline is neat and speeds up simple scenes. It doesn't really speed up *complex* scenes that much, and, also, creates some memory consumption that isn't flushable.

 

Lemme know the memory consumption after doing that...

 

/Z

 

aaaaah i already write that! ZAAAAAAAAAAAAP

ehehe

[Crazy Homeless Guy]

Switch over to BSP2. I'm not sure why many people are overlooking this, perhaps it wasn't focused on enough in the documentation? BSP2 is much more efficient at handling millions of instances.

 

Bottom line, when using a lot of instanced proxy objects, you should be using BSP2 for optimal results.

 

I think this may be missed by newbies such as myself because it is off by default. I wouldn't know to switch it on, and if I came across it, would assume it was going to increase the rendering time of the image.

[Crazy Homeless Guy]

 

 

The color and contrast of this test is nice.

[dagon1978]

The color and contrast of this test is nice.

 

thanx

for what i can see, in your scene there is something more to change, for exteriors you can use burn highlight=1 in the exposure control, this means richer colors (less compression) and higher contrast

and for the FG, right now i dont think the old radius method is a good start point, try the new interpolation instead and use the exact AO in the arch&design to get the details

[pixelperfectg]

oh, i added your site and blog on the links page, are very insipring for all the mray community... i hope is not a problem for you

No problem at all, thanks for the addition.

 

I think this may be missed by newbies such as myself because it is off by default. I wouldn't know to switch it, and if I came across it, would assume it was going to increase the rendering time of the image.

Exactly. Evidently it's not featured prominently, clearly, or in the right location in the documentation because I know several users that saw the new BSP2 option, but didn't know when to use it (or even if they should). Same holds true for disabling the scanline mode.

[Justin Hunt]

Funnily I have been wondering what exactly scanline is handling, sad to say I couldn't be bothered to RTFM

 

I am rendering off an animation with several different proxies instanced many times around, other than a few unexplained crashes I am very please with the results.

 

crazy homeless guy - Did you ever find out what the shadow problem was?

 

jhv

[Crazy Homeless Guy]

thanx

for what i can see, in your scene there is something more to change, for exteriors you can use burn highlight=1 in the exposure control, this means richer colors (less compression) and higher contrast

and for the FG, right now i dont think the old radius method is a good start point, try the new interpolation instead and use the exact AO in the arch&design to get the details

 

 

care if i ask what your sampling and filter settings were for this image?

 

crazy homeless guy - Did you ever find out what the shadow problem was?

 

i know in my other post i had a few trees floating. those were a result of poor proxy creation by myself. the pivot was centered, so when i scaled them the smaller ones were floating.

[dagon1978]

care if i ask what your sampling and filter settings were for this image?

 

it was a test, so i was using min 1/4 max 16, mitchell 1-1 (this is not usefull in production, you have to use lanczos 4-4 if you want a sharp image)