I do it quite often but it's usually just a few rocks at a time. Either way if you don't do it that often your most likely to stress your fish out. I'd do a water change afterwards cause you may stir up detritus, but I don't know anything about your tank but you'll be able to tel once you start. If you do decide to catch your fish first than take some water out of your tank for them first before you start mucking around. Other than that just try not to squash your fish
