Alpha white matte / halo / fringe / outline

[johnharrison]

Hi. I know this have been discussed several times but after A LOT of googling on the topic I find myself no wiser. Say you want to render an arch viz image with a hdri map, and replace the background in photoshop with a better suited image. I have found this problematic as there are white outlines when the alpha is applied in compositioning.

 

I use 3ds Max, vray, photoshop and I usually render to .tif files.

 

Does anyone have any solution?

Edited April 16, 2013 by johnharrison

[Francisco Penaloza]

Hi,

in photoshop you select your layer, then in the top menu Layer-> Matting-> defringe, you can adjust the pixel size but usually 1 or 2 pix will do the trick

Other way is using remove black matte or white matte, but defringe does a good verage of both so that should be your new best friend.

Cheers.

Fco.

[Chris MacDonald]

I was going to say Defringe too.

[johnharrison]

Hi, thanks but I've tried those before and they rarely seem to work. Especially defringe is sketchy, seemingly stretching the pixels vertically. See image attachment.

 

[notamondayfan]

When you render your images, is the HDR visible, or is the background black? Ideally you should render the background black for compositing.

 

Dean

[John Dollus]

In Pshop, create a layer mask using the alpha. Right click over the mask and select 'refine mask'.

[johnharrison]

EDIT: Both defringe, which turned out with quite horrible edges, and remove black/white matting seemed to work properly when first applying the alpha to the image.

 

When you render your images, is the HDR visible, or is the background black? Ideally you should render the background black for compositing.

 

Dean

 

I usually, if a backplate replacement is intended, render with a black background, yes.

 

John Dollus; That didn't seem to work very good. Which settings do you usually shift around with?

Edited April 16, 2013 by johnharrison

[John Dollus]

shift edge in a negative amount works like defringe but with more control and allows for non destructive refinement since you don't have to actually apply the mask to the layer. decontaminate colors can also help with background color artifacts.

It also works well for painting in varying amounts of refinement in situations like tree leaves or railings.

defringe/remove matte is fast and easy but it can be heavy handed sometimes and is a destructive process since it has to be repeated if the layer updates.

here's a good intro to how to use it:

http://www.adobepress.com/articles/article.asp?p=1950633&seqNum=4

[johnharrison]

Thanks for the tips, John. Although I believe it can be very effective in alot of situations, I found it to be "damaging" to my scene as it made the curtains who are somewhat transparent, very transparent. (See curtains in post 1)

[johannestiner]

Save as 32bit exr and choose transparency instead of alpha when importing the image to photoshop is the only way i have found to work without having to do any work.

 

If that is not something you want to do then render your image with the hdr or similar as background, if you dont want the background to affect the light and reflection just use the environment override and set it to black. You will still get halo but it wont be visible since your using the same or similar hdr as background.

 

You can also tweak the alpha with the mask options or blurring the mask and use the level adjustment.

[johnharrison]

Thank you (Tack så mycket ) Johannes. I'll try this. Is saving to .exr files a common practice? As far as I've understood, alot of people use tiffs when saving their renders.

[johannestiner]

Exr is becoming the standard, or it is the standard, this is at least what i think and exr is pretty sweet since you can have the passes in one file and deep compositing (maybe the deep comp work in other file extensions)

 

You can try and save the tiff in 32bit and see if it works, i have just tried with exr.

[johnharrison]

Ok, I've been looking into the EXR file format, and it's still very confusing. I say still as it's not the first time I have done so, and I gave it up back then. I was wondering if you could perhaps write a little guide on saving the files in max and opening them in photoshop, and other common issues for a first time user?

 

Do I use ProEXR from fnord? And do I use ProEXR or ProEXR EZ? I dont understand the difference, and it seems the EZ version is the only one with the option of importing the exr with transparency instead of alpha as you stated in a previous post. Also have a look at my attached image. Im thinking I might aswell export as 16 bit as photoshop dont really support 32 and I usually wont need the extra information per pixel?

 

Also; do you save the render as an RAW file, and then specify it as exr or do you save as split render? ( both in the vray frame buffer tab)

 

Edited April 18, 2013 by johnharrison