VRay render with alpha contains a white matte/outline

[ryandmonk]

I'm trying to render a very simple animation with an alpha channel so I can do video composting to it. I'm using 3ds max 2010 SP1 with VRay Adv 1.50.SP4a. The issue that's going on is that there's a 1-2 pixel white matte/outline the geometry edges because of the alpha channel. I'm rendering as a targa sequence with a split alpha.

 

Other areas I've played around with is the the material window, where I made sure the "Affect channels" is set to "Color+Alpha", but overall I'm trying to figure out how to make the alpha channel more precise without the couple pixel edge.

[Nic H]

are you rendering on black with a pre-multiplied alpha?

that should fix it.

[ryandmonk]

i had VRaySky as my environment background. I disabled it, and made the background color gray. The outline is still there and visible if I make the background color to a black, white, etc.. but the gray hides it well.

Edited January 14, 2010 by ryandmonk

[Nic H]

hmm is it something to do with using adaptive subdivision AA maybe? or clamp output subpixel mapping?

[ryandmonk]

the AA filter is "Area" with a size of 1.5 and sub-pixel mapping is turned on. i also previously tried turning on clamp output with the same effect.

 

the adaptive subdivision image sampler setter is: min. rate of -1, max. rate of 2. clr thresh is at 0.1.

 

other people have also mentioned several times about making sure sub-pixel mapping is turned on, but i'm thinking this cause is from something else.

 

if i switch to another renderer, such as mental ray, the alpha channel is dead on, so i know its something with vray...

[Nic H]

hmm not sure then sorry

i dont use the adaptive subd AA but is it object outline being unticked?

[Nic H]

maybe just maybe this might work!

[ryandmonk]

...sigh... it was unchecked. i checked it as well... same results. ill learn to deal. thanks.

[Ricardo Eloy]

Try saving as PNG with Pre-multiplied Alpha unchecked. See if that works.

[ryandmonk]

nope

[Sassmouth]

Ryan,

 

Did you ever solve this issue? It's plagued me off and on for years, and I've never been able to definitely solve it. It's like a 1-pixel-wide white outline, and so annoying.

 

Thanks,

Steve

[bluetorch]

Same issue, can't solve it either...

[Devin Johnston]

I don't know what causes it but if you're using after effects to comp the video you can use the Remove Color Matting effect to minimize the halo around your objects. I'd really like to know what causes it as it makes working with layers very difficult, have you posted your issue on the Vray forum yet?

[dmv79]

This is how I get around the issue:

Before rendering, I try to match the background color with whatever I'll comp it with... If I'm not sure then I go with black.

Save the alpha channel as well.

In PS, apply defringe to the alpha map (most cases) and mask all the rendered layers with it.

In AE, use the same alpha map (most cases with color matting) set to "stencil alpha", thus masking all the rendered layers.

All this may be dumb solutions... but who knows? Works for me.

Worth mentioning that higher resolution renders and AA settings will incrementally help a lot.

[Devin Johnston]

There's a thread on the Vray forum that's talking about a similar issue, the consensus is that Vray seems to be getting worse in dealing with render channels. There's obviously a problem and it's been there for a while, I don't know why it's being ignored seeing as how it's pretty important.

[erickdt]

Although it's not a solution to the problem (that ball is in Chaos Groups court) you can manually adjust clipping masks in Photoshop by right clicking on your layer mask. There all sorts of settings you can tweak to reduce the ammount of fringe.

 

E

[3dterior]

I always got this problem when using vraysun+sky in exterior scene. I think the problem is glossiness from material that reflect vraysun's light is cause of this outline effect.

The solution to solve this problem that I always use is putting something (such as trees or buildings) behind the scene to block vraysun's reflection from creating glossiness to the edge of object.

[Alyosha]

I've pretty much managed to contain this problem by using backgrounds that roughly match the background used in the composition and using unmultiplied TGA file.

 

Also when doing videos, tools like after effects have advanced tools that allow you to trim off these kind of things without doing too much damage to the render.

[Origins]

i think the AA cant handle 32bit depth so i use max standart cameras/low intensity light setups (example: vraysun=0.01 instead of default 1.0).

 

20+ animations and 100+ renders so far...

[Origins]

OKay , found another solution.

 

 

check "dont affect colors" , clamp, subpixel mapping on. now u can use hi intensity light as usual...

 

the downfall is you wont be able to use reinhard or anyother mapping except linear.

[satyanreddyg]

me too same problem

[fanisanastasiadis]

Hello!

I had the same issue when rendering grass, I think I may have found a solution. I remade the alpha channel pic many times and nothing changed. This happened only for grass and not for leaves on trees. The difference between the two : leaves have black all around them while grass touched 3 sides of the alpha. So i cut a very small black outline around the alpha for definition and it worked. I think however small its allowed without compromising continuity will work fine.