I think I've figured it out! (seconds after posting the above question, but hours after searching for how to do this correctly ).
1. Under reflection map with TexFresnel chosen - Under the map for (reflection) IOR enter your transparency mask image. (don't touch refraction IOR)
2. change the color of Perpendicular to something other than white if you wish to reduce the reflection.
Can also be done without the Fresnel:
1. Under the map for reflections choose TexBitmap. Enter your mask image in there.
2. change the amount of reflection by changing the number beside the map.
Hope this helps someone
