I don't know if this will help with your issue, but in the past when I have had weird results with glass refraction and reflections I found that upping the max depth for the refraction got rid of many different issues. Might help here with so much glass if you have it set at 5 or even just 10. I've upped it to 20 or 25 and gotten the right result without crazy render times.