Cheat, try applying a grey material to the grass object before doing the irradiance solution, then save your solution. and then apply the grass material to the grass object and render again with the saved solution, that way the grey material will be baked into the building and not the grass material. Please correct me if I'm wrong.