Compositing in 3ds Max and V-Ray

[braddewald]

I am fairly new to architectural visualiization (3ds Max + Vray) and somebody told me that rendering in separate channels like shadows, GI, Ambient Occlusion etc. would allow me to have more control over my final image. I understand the concept but not the execution. Does anybody have any advice or know any tutorials that cover compositing in 3ds max and photoshop so that I could better understand how it works? --Thanks

[WAcky]

Hi Brad,

 

I've recently been looking into this and it would appear that this is a great workflow but unfortunately photoshop is just not equipped with the right tools to come out with a flawless recomposition of a render thats been split into elements.

 

This comes down whats called pre-multiplied images and when it's not dealt with properly, can created halo's around objects. Unfortunately photoshop doesn't AFAIK have tools to "un-pre-multiply" and pre-multiply images.

 

It's hard to explain without visual cues but a node based compositor is essential to this type of workflow. It opens everything up and makes life a lot easier.

 

Programs that are capable of this workflow are softwares like Digital fusion and Nuke. If you are thinking of doing any animation down the track it could be an opportunity to start learning this software as they are geared towards animation.

 

Here is a DVD aimed at compositing 3ds max elements in Digital fusion

Here is a great tutorial aimed at compositing Vray elements in Nuke

 

I think it's a workflow that has benefits but I doubt I'll be looking into this workflow until I get into animations.

[dean@pikcells]

We have used photoshop for our compositing and post processing for many years, and dont really have any issues. Although recently we are looking to use After Effects for compositing the passes. The main issue with photoshop is that it doesnt support floating point images (16bit), where was After Effects does.

 

We render all our images as passes, and you are correct, it does give you so much flexibility. Changes to materials, lighting, etc can sometimes be a very simple tweek in post production, rather than wasting time rerendering your scene again.

 

The Gnomon DVDs are a great place to start, even if you dont use the exact same software. I have watched the compositing in Fusion, and a lot of the work-flows, and understanding, can be used in most applications.

 

As for the pre-multiplied issue, there are several ways around this, but if you want to do things correctly, use a compositing program like Fusion, Nuke or AE. I think Fusion also has a free learning edition too.

 

Hope that helps, Deano!

[Devin Johnston]

Fusion and Nuke are both $4,500+ programs while After Effects it around $1000, if you want to start on composting I'd start with the cheaper option and work my way up.

[Jimy]

We use Fusion/ Shake generally and then take the comped passes into Photoshop to finish off stills. WAcky is completely right in that Photoshop does not deal well with pre-multiplied alphas and after you comp all those passes you'll end up with a faint black line round everything which looks a bit rubbish.

 

However it is possible to comp these passes in Photoshop if you use the defringe or remove black matte under layer>matting>defringe with a good degree of success. If using the black matte though you must render with a black background for this to work properly.

 

You can then stack all your render element layers as clipping masks on the top of your base/ beauty pass. (alt-click on the line between your layers, or layer>create clipping mask)

 

Hope that helps.

[Jimy]

The main issue with photoshop is that it doesnt support floating point images (16bit), where was After Effects does. Deano!

 

Hey Deano, I think Photoshop does support 16 and 32bit images (image>mode)? We use 16 and 32 bit occasionally, and I definitely use it for comping RAW photography. Unless I'm mistaken on what ya mean

[rmccoy]

This has been discussed some before. In this thread there is a breakdown of what you need to render out from Max with Vray and how to reassemble them in Photoshop.

 

http://www.cgarchitect.com/vb/35292-compose-rendered-elemnts-photoshop.html

 

This is what all you need (taken from thread mentioned above)

 

Composite Breakdown for Vray (as posted by others)..... (Add) = Linear Dodge

 

Vray Specular (Add)

Vray Background (Add)

Vray Self Illumination (Add)

GROUP (Add)

___Vray Raw Refraction (Multiply)

___Vray Refraction Filter (Normal)

GROUP (Add)

___Vray Raw Reflection (Multiply)

___Vray Reflection Filter (Normal)

GROUP (Add)

___Vray Raw Global Illumination (Multiply)

___Vray Diffuse Filter (Normal)

Vray Raw Lighting (Multiply)

Vray Diffuse Filter (Normal)

 

Base of any scene, regardless of engine...

 

Vray Specular (Add)

Vray Self Illumination (Add)

Vray Background (Add)

Vray Refraction (Add)

Vray Reflection (Add)

Vray Global Illumination (Add)

Vray Lighting (Normal)

Edited September 15, 2009 by rmccoy

[Jimy]

Hmm, that's a new way of doing it. I must admit we find the following pretty reliable, but I'm definitely gonna check out that thread to see what it offers.

 

(nb adjust opacities of these layers to suit image)

Vray Specular (Screen)

Vray Reflection (Screen)

Vray Global Illumination (Screen)

Vray Lighting (Screen)

Vray Shadows (Screen, will cancel shadows- use to remove shadows)

Vray Shadows inverted in PShop (Multiply, will add shadows)

VRay Extra Tex with Dirt in the channel (Multiply, or use as a mask on adjustment layers to saturate colours in shadows and reduce levels to darken corners)

 

Give this a go and see what you think... I should really blog this with examples!

[braddewald]

Hey rmccoy, I was just waiting for this suggestion to be posted on the thread, haha. I saw it a while after I started this. I have been trying this technique all day today but just can't get it to look the same. The composite looks overexposed and generally poor compared to the regular rendering. Can you offer any advice on why this may be happening. Could it be because I am using an old version of Photoshop (Photoshop 7.0)? I followed the instructions exactly. I also turned off clamp output and sub-pixel mapping (which I usually have on) and saved each element as separate jpegs and compiled them as layers in Photoshop.

[rmccoy]

it could be the fact you are using jpgs. I always saved them out as tiff's. That way there is no compression on the individual layers. I'm not really sure why they are not coming out the same. If you are doing the basic version, you should add the vray specular element to it also. put it on top and change it to additive mode. I tried this method using both photoshop and after effects cs3 and it worked out fine.

[braddewald]

So rmccoy, I have attached a screenshot of the problem I am having with your setup. I just can't get it to work (but I am following it exactly, at least in terms of Add (linear dodge), multiply, normal blending and so on). The only thing I can think of is that I haven't jumped on the gamma 2.2 bandwagon and its causing problems. The image on the left is the regular pass and the right side is the composited image. WHAT AM I DOING WRONG?!

[Jimy]

Hmm looks as though those Raw passes are over-cooking your scene. Just for comparison try what I posted before (below) with output to targa files (compress them if you like). Honestly we do hundreds of stills and films using this method and it always works...

 

(nb adjust opacities of these layers in PShop to suit image)

Vray Specular (Screen 100%)

Vray Reflection (Screen, 60%)

Vray Global Illumination (Screen, 10%)

Vray Lighting (Screen 100%)

Vray Shadows (Screen, will cancel shadows- use to remove shadows

Vray Shadows inverted in PShop (Multiply, will add shadows >100%)

VRay Extra Tex with Dirt in the channel (Multiply, or use as a mask on adjustment layers to saturate colours in shadows and reduce levels to darken corners)

[rmccoy]

That's really strange. Do you have the groups set to the correct blending modes? If this still won't work I would either try the basic method or the method that James has listed above. Would you be willing to email me the different passes so I can give them a shot?

[braddewald]

James, I tried the method that you listed and it seems to be working ok. Ross, your method seems to allow more control if I could get it to work. I am absolutely willing to send you an email with the passes. Just send me a PM with your email address. Thanks all.

[Nic H]

best thing to do is use your eyes.

 

and experiment, people get caught too up in all this science crap and lose track of trying to make a good image

Edited September 16, 2009 by nicnic

[Q-Bix]

I have yet to try this technique. What are the main advantages? Is re-rendering one pass ie. diffuse only, quckier? or is the flexiblity within photoshop itself?

[tayrona]

best thing to do is use your eyes.

 

and experiment, people get caught too up in all this science crap and lose track of trying to make a good image

 

As always.. a "good landing" point.

[arturito]

Vray Specular (Add)

Vray Background (Add)

Vray Self Illumination (Add)

GROUP (Add)

___Vray Raw Refraction (Multiply)

___Vray Refraction Filter (Normal)

GROUP (Add)

___Vray Raw Reflection (Multiply)

___Vray Reflection Filter (Normal)

GROUP (Add)

___Vray Raw Global Illumination (Multiply)

___Vray Diffuse Filter (Normal)

Vray Raw Lighting (Multiply)

Vray Diffuse Filter (Normal)

 

hello everybody, I use max and vray and actually I need to start working with elements, photoshop composition etc. I am following this Breakdown in a test scene and I have a first question: When you write in the list:

 

GROUP (Add)

___Vray Raw Refraction (Multiply)

___Vray Refraction Filter (Normal)

 

It means that you merge these two layers and then Add the result? what means exactly GROUP in photoshop?

 

Thank you very much.

[WAcky]

I think it means you put the two elements into a folder and set the folder's blending mode to "add" in this case.

[arturito]

Thanks 4 your response Wacky, Hmmm I am trying, but when I make a group, then the Blending options does not appear in the right click menu as it appears on standard layers.

 

I tried merging both layers and then "add" the result and I think is working. I will keep on playing. And asking I guess.

[WAcky]

No, dont merge the layers.

 

Instead of rightclicking the folder, look just above the layers pallette. There should be a dropdown menu and if you have a folder selected, it should say "pass through". Change this selection to "linear dodge (add)"

[arturito]

Ok I finally got it (thanx wacky, not only for this post, but also for all tips you usually share). I am posting three images and attaching my test scene (max and psd) so it could help newbies like me to understand the proccess.

 

I have some [beginner to intermediate] questions

 

1. I had to turn off the Raw Reflection filter to achieve the exact same look of the original render, is it normal? (as you can see in my images, with Raw Rflection Filter ON, glasses look darker)

 

2. I used TIFF 16 bits images, is there a better format option?

 

3. The composition looks with a thin black line in the edges, is there any tip to avoid this?

 

Thanks!

[WAcky]

Thats interesting how you turned off that layer and managed to get it the same as I'm aware a correct comp doesn't acheive the 100% match to max's output.

 

The small black lines/edges is a result of photoshops inability to correctly handle pre-multiplied (anti-aliased) edges perfectly. Unfortunately this will always be present unless they make photoshop better at this operation. The only way to avoid this is to render at ever higher sizes to minimise the effect, or start using a better comping program like fusion or nuke.

 

edit: you might want to play around with the layers>matting options for the outer ring. I'm not sure how your file is setup but this is how I get rid of my halos. I render to a black background and use the remove black matte option which seems to work pretty well (for photoshop.) If you don't use a black background you can use the defringe option but this doesn't play well with transparencies.

Edited November 23, 2009 by WAcky

[arturito]

Matting option sounds better than defringe because I prefer using black backgrounds. I will try it and submit results. thanxs again

[braddewald]

Does anybody know how much adding render elements affect render times?