I have had this problem since I started using HDRI's for GI.
I have had this problem since I started saving files in a High Dynamic Range format. The edge of the model does not blend/AA smoothly. I get a funny line. If I save the file in an 8 bit format, there is not a problem.
If I want to do post process depth blur, etc... this becomes a problem. The only way I can think of around it would be to render wioth a sky dome, so there is no 'edge' to the model.
...but I was under the impression that I shouldn't have to do this. I should be able to use a HDRI image for my GI, and also assign it as the enviroment map, and have a seemless image.
I posted a similar question about a problem with AA around Vray Light materials a couple of days ago. Someone responded by saying that the High Dynamic Range format was causing this because of the light intensity at the efge of the material. It made sense, but I felt it was crap as far as the file format goes. A weekness that there has to be some way around, but that I don't know as of yet.
So, any ideas.....
I am having the same problem with HDR, EXR, RPF formats.