Question for fell Onyx/3ds Max/v-ray users

[Kawzy]

Okay, so, I apologize if this has been discussed, but I couldn't find this specific problem by search, so I figured I'd just post. I've been experimenting with using the vray 2-sided material with bitmapped leaves on an onyx tree, similar to the tutorial that was posted on here not long ago. (http://vray.info/tutorials/leafmaterial/) The problem I keep running into is that when the leaf texture (.tif with an opacity map) has sky behind it, there is a dark halo effect. If the leaf is in front of geometry, this problem doesn't exist. I was wondering if any of you out there have tried the tutorial as well and run into a similar problem. If so, has anyone found a solution?

 

The below image represents what I'm talking about. I've turned off stems and twigs so the leafs aren't really floating, I'm just not displaying the geometry in the tests. The maps are all 8-bit .tif files. I've tried adjusting the blur settings, and this hasn't helped. I know it's a bit difficult to see the halo in the low rez image, but trust me, it's there

Edited March 29, 2010 by Kawzy

[Kawzy]

Sorry. Forgot the attachment...

[Peter M. Gruhn]

Does it happen with a generic material of a similar color?

 

Are you using a filtering option with a lot of sharpening?

 

Sorry, all I can think of right now and can't test (heck, don't have vray at all ;-)

[Kawzy]

If I use a normal v-ray material instead of the 2-sided technique I don't have the halo issue, so I somewhat feel that it has to do with the translucency, but I followed the tutorial that I posted a link to, so I can't figure it out. His looks fantastic.

If I experiment with the image sampler, the less sharp filters do indeed produce less of a halo, and it's almost not even visible. But, it is still there. I even went into photoshop, took the original leaf material and changed the background color to a greenish tone and added a blur between the leaf and the background, so even if the alpha isn't perfect, it should still be returning green instead of black. Also, it seems that the halo is more pronounced when you see a leaf from the back, also leading me to believe it's due to something with the translucency. But I can't for the life of me find out what, and it's starting to drive me crazy. The tutorial is definitely something that we'd like to incorporate 'cause it looks pretty.....I'm probably just missing something small and was hoping someone would just say "duh, change (insert extremely easy fix here)"

[Peter M. Gruhn]

He makes a point of saying it is important that the opacity map have a sharp edge. That sounds like black OR white only. The sample image appears anti-aliased, but maybe that's because he resampled down for the web?

 

What does your translucency map look like?

[Kawzy]

It is indeed due to the web. It's rendered at a lower resolution and I've zoomed in to try to show the edge. And yes, the sharp edge is desired. I'd rather not settle with just a blurred edge if I can avoid it, but if that's what it comes down to then I guess I'll settle.

As far as the translucency map goes, I first tried using a solid color thinking that the opacity map would cut it out where needed and save time over making a new map for each leaf type of each different tree we'd eventually use. I tried pure white and various tones of grey.The translucency worked fine, but the halo occured.

So next, I went into photoshop and created a custom map, somewhat immitating what was shown in the tutorial. There was no visible change. I also experimented between changing the color swatch to the left of the translucency to multiple ranges between black and white. I've posted the current leaf setup so you can check out the maps and see if anything looks off.

[Peter M. Gruhn]

Afraid I don't see anything obvious (but you knew that already ;-). My next move would be to start playing with them, but I worry that it's a renderer specific issue and I may be able to make a great version with no issues in MR because I don't have the VRay "Break Leaves" button.

[Maestro101]

Hey. I have the same problem. I think that maybe its becouse your leaves are not planes, but have other shape. As far as I see, Onyx generate leaves which are similar to the shape of true leaves. I didn`t check that hypothesis yet anyway.

[Jimy]

Are you putting a background in in comp? If so are you rendering on black? If you render on grey like the left hand side of your image and you pre-multiply that on top of your background you will get a halo, as when you multiply grey with alpha you get a colour, unlike black which is value 0, so you get 0, or full transparency.

 

Try rendering on a pure black background then pre-multiply over your background image.

[Kawzy]

Hey Maestro. It's true that Onyx creates leaves with a shape to them. In this scenario however, I'm using TreeStorm to bring the tree into max, and the leaves were converted to plates and then opacity mapped. So, technically, they are planes in the view. I even removed any resolution, so they don't have any real shape to them.

 

Jimy, the background is integrated through Max here, and the grey on the left of the image is just a piece of geometry with a flat material on it so you can see the black edge only occurs over the background. I took a shot at trying to add one in post just to test it, assuming that it would be worse, and it was. The black halo wasn't there, but as expected, you actually get a worse halo effect of a lighter color. I'm assuming becasue of the translucency being used, you get extra color data that just doesn't jive with any background that doesn't match your lighting. And part of the desired look using translucency is to be able to pick up a little of the blue color of the backround sky through the leaf, which got lost somewhat when I rendered on black and composited. The key I've found is just to not use any filtering, regardless of how you want the edges to look.

 

The problem from what I've been able to determine lies in the filter. It seems that translucency doesn't play so nicely with it, and you have to just disable it. Peter hit it pretty much in the beginning and knew it was the filter. Guess I should've just listened instead of stubbornly pursuing my clean-edged leaf dream while using one. Not sure what everyone's work flow is out there, but it also helps get a cleaner edge if you enable the sub-pixel mapping option in the Color mapping rollout. Just updating anyone that my read this. And if anyone else out there comes up with something else, I'd love to hear it.

Edited April 9, 2010 by Kawzy