If you want the flute to remain your main instrument then it is up to you that you make sure it is. Don't spend more time on the saxophone and neglect the flute.
I don't really know about the effect on tone as I went from Sax to Flute, but I am managing perfectly fine.