Is it a bad thing.. ?

[Delphinus]

My tang goes nuts for that stuff.

I've never really concerned too much over it because I figure my skimmer will pull it out or my tang will eat it, or I'll siphon it out during a water change. If you did want to remove it though, I imagine if you could shut off your pumps during the scraping, you could probably suck the most of it out with a turkey baster.