My yellow coris picks at them...but I think it's just out of boredom. I usually just see a couple floating around the tank, and I'm sure they eventually reattach somewhere. My emerald crab used to eat them...but he also picked at the polyps on my sps so he was banned from the tank.
I think it's hit or miss and you'll end up with a fish or invert that's "reef safe with caution", and thus run the risk of them picking at or eating coral or other inverts as well.