You can still bring your po4 down just do it slowly , keep testing for your nitrates and wait till they are detectable then up the gfo Amount.
You'll really need to consider how you will keep nitrates low long term though , many choices to choose from.
Keep your parameters stable throughout all this and your system will adjust to the nutrient changes rather quickly , quick drop in nitrates usually isn't an issue so I dont think its the exact cause but sure isn't helping.
__________________
........
|