Alpha lunacy

[Karl Larsen]

Can anyone explain, lucidly, and sequentially, with the appropriateamount of relative detail, without including a bunch of irrelevant stuff, how to create a transparent alpha channel and get it to work in 3DS MAX MentalRay? (the concept should be the same in Adobe Photoshop or Corel PhotoPaint – I have both, I prefer Corel).

 

I made trees in Xfrog. I know how to make/get leaf image files and apply them in Xfrog, IF I use a .tif leaf image file that came WITH Xfrog, everything works fine...

 

I photographed and scanned a leaf image from my yard, and not for lack of trying but I cannot figure out how to get the fricken alpha channel to work!!!!

 

2-nutty things:

 

A. When I import an Xfrog tree with Xfrog leaf .tif, that I modify in Sketchup – 3DS Max seemingly converts the leaf .tif files to .png’s and everything works perfect with NO WHITE HALOS. Is that (convert to .PNG) normal?

 

B. When I import and Xfrog tree that I made with a.tif file I created (that supposedly includes an alpha channel), it either has NO alpha channel or is has a white halo that makes me want to find the programmers that made all this stuff and drown them.

 

(.PNG's just have that horrible white halo)

 

I tried VRAY for Sketchup and, it’s cute, and easy, IF you're making cartoons and/or a scene doesn’t exceed kid-cad MB’s/polygons (save to VRIMG does not work as advertised if you have a 300MB scene – which, paradoxically, turns out to be only about 160GB after importing it into 3DS - go figure). So I am back to modeling in SketchUp and importing the SKP’s into 3DS Max Design 2013… It’s doesn’t care how big files are.

 

I just want the fricken leaves to look right!

 

Thanks in advance.

 

High Current Pro HCP 750

ASUS Sabertooth X58

I7-980 Gulftown 3.33 GHz 130W 6-core

CORSAIR Vengeance 24GB

ATI Firepro V7800 2GB

Intel 520 Series 240 GB SATA III SSD (os/programs)

Intel 320 Series 160 GB SATA II SSD (data)

WD VelociRaptor 300GB SATA II HD (data)

Maxtor 160 GB SATA HD (data)

MS 7PRO 64

HP ZR24W LCD

 

I'm pretty sure my computer can handle it...

Edited August 8, 2013 by karlar

[notamondayfan]

For starters, your PC is fine, it's a software issue

 

The haloing around maps occurs because Max (and other software) blurs the maps inside the software. By blurring the maps, you will get haloing. This is because the alpha map instead of having a crisp black and white edge, will be turned to a blurry edge going from black to grey to white.

 

First of all, I wouldn't embed alpha info into map files. 99% of my maps are jpgs, because of the file size and flexibility. In vray instead of use png or tiff files with alpha, I would have 2 maps, a diffuse and an alpha map. The diffuse goes in the diffuse slot, and the alpha goes in the opacity slot.

 

The quickest option to reduce haloing is to turn the blur value of the maps down in the bitmap settings in the bitmap slot. Sometimes this will work fine.

The other way is to modify your diffuse image. Where your leaf edge is, paint, clone, or what ever to make the leaf larger and overlap the leaf edge. This is so that when your alpha map is blurred in the software, the gradient edge is still there, but as the edge of the leaf covers the gradient you won't see any haloing.

 

I hope that helps, I'll see if I have time later to do an example if you're still struggling.

 

Dean

[Corey Beaulieu]

Have you tried to just make a black and white map of your alpha channel in PS and then using that in the Opacity/Cutout/Transparency slot of your material?

 

*Dean, your response came in while I was typing. Didn't mean to reiterate what you already said.

[Karl Larsen]

Dean,

 

I owe you one! That worked perfect! It's a wonder information like that isn't more readily available...

 

I hope I NEVER have to deal with an alpha channel ever again!

 

Thank you,

 

For starters, your PC is fine, it's a software issue

 

The haloing around maps occurs because Max (and other software) blurs the maps inside the software. By blurring the maps, you will get haloing. This is because the alpha map instead of having a crisp black and white edge, will be turned to a blurry edge going from black to grey to white.

 

First of all, I wouldn't embed alpha info into map files. 99% of my maps are jpgs, because of the file size and flexibility. In vray instead of use png or tiff files with alpha, I would have 2 maps, a diffuse and an alpha map. The diffuse goes in the diffuse slot, and the alpha goes in the opacity slot.

 

The quickest option to reduce haloing is to turn the blur value of the maps down in the bitmap settings in the bitmap slot. Sometimes this will work fine.

The other way is to modify your diffuse image. Where your leaf edge is, paint, clone, or what ever to make the leaf larger and overlap the leaf edge. This is so that when your alpha map is blurred in the software, the gradient edge is still there, but as the edge of the leaf covers the gradient you won't see any haloing.

 

I hope that helps, I'll see if I have time later to do an example if you're still struggling.

 

Dean

[Karl Larsen]

What Dean said!

 

Thanks for the quick replies! I thought I was going to lose what little mind I have left!

 

 

Have you tried to just make a black and white map of your alpha channel in PS and then using that in the Opacity/Cutout/Transparency slot of your material?

 

*Dean, your response came in while I was typing. Didn't mean to reiterate what you already said.

[notamondayfan]

It's just one of those things that can drive you insane, but it's so simple to cure when you know how!! Anyway glad I could help.

[Karl Larsen]

I would still like to learn how to create the alpha channel and get it working for .tif images because - if you apply, for instance, a .tif leaf image in Xfrog (presuming it has a working alpha channel), when you import the tree/plant/shrub into 3DS everything is already done for you (I wouldn't have to apply a diffuse and/or a cutout for the leaves).

 

It would make life a little easier, but at least I can get by for now.

[notamondayfan]

I'm guessing if you expand the edge of the diffuse part of the tiff, and left the alpha channel as is, then that too would stop haloing.

[Chris MacDonald]

Disabling filtering/blur on the alpha channel will also massively speed up render times (in vray at least). Just covered it in my last blog post.

[Nils Norgren]

The absolute best way to use embedded alpha channels is to use the "Straight" alpha channel approach, (vs "premultiplied") this will work with all the filtering and blurring in their default settings. These are old-school problems that people ignore these days but it is still valid to understand and use these terms appropriately. A straight alpha channel has an RGB channel with extra pixels around the edges, so that when transparency is introduced through the alpha channel, it is not grabbing (in this case green) wanted color mixed with unwanted background color (in this case white).

 

The short answer is, make the alpha channel have all the transparency, (clean edges with anti-aliasing, black and white with grey edges for transparency) and the RGB channel should have extra color around the edge of the leaf (or make the whole background green). When importing the map, turn off the "Premultiplied" check box on the importer.

 

If you want to stick with defaults then be sure to use a black background in your RGB channel, then the file is "Premultiplied" and will attempt to subtract the black out of the edges when it reads the bitmap's transparency.

 

As I stated, these problems don't rear their heads as often as they did in the old days, but it is still relevant information to understand alpha channels and edge treatment. This is very important when comping and dealing with passes in a comping program. I am sure many have seen the blue fringe when using a render in AE and are replacing the sky.

 

http://help.adobe.com/en_US/AfterEffects/9.0/WS3878526689cb91655866c1103906c6dea-7f7ea.html

 

Hope this helps - Nils

 

Just found this: http://forums.cgarchitect.com/61108-white-fringe-ring-when-compositing-trees-exr.html

Edited August 9, 2013 by NilsN

[Karl Larsen]

How to you MAKE a transparent alpha channel in a .tif, in the first place! Once I get that working I'll focus on the edges.

 

You previous suggestion worked perfect - I created a diffuse .jpg and an inverted b&w of the same image for the cutout = NO halo!

 

I'm guessing if you expand the edge of the diffuse part of the tiff, and left the alpha channel as is, then that too would stop haloing.

[Nils Norgren]

Not sure about corel, but in photoshop there is a "Channels" UI element (usually buried behind "Layers") here you can isolate the Red, Green, or Blue channel of an image, to make a new channel (Alpha) you select "New Channel", and then paste a Black and White image into that channel that represents the transparency of the image(white is opaque, black transparent). An alternative is "Save Selection" which achieves the same thing, useful if you have a selection representing the transparency.

 

-Nils

[Karl Larsen]

Now I think I am getting it!

 

Previously (according to the Corel instructions) I had been adding a 'mask' to the alpha, NOT a b&w image - I believe you may have solved the puzzle for me. It makes sense to me now because (reading from the beginning of this thread) all of the Xfrog leaf .tifs, have inverted b&w .tif versions of each leaf image, which must have been used to create the alpha in the diffuse image, and I wondered what in the heck the b&w images were for

 

 

Not sure about corel, but in photoshop there is a "Channels" UI element (usually buried behind "Layers") here you can isolate the Red, Green, or Blue channel of an image, to make a new channel (Alpha) you select "New Channel", and then paste a Black and White image into that channel that represents the transparency of the image(white is opaque, black transparent). An alternative is "Save Selection" which achieves the same thing, useful if you have a selection representing the transparency.

 

-Nils