I also do this, and it does affect speed - one way to start correcting it is to look at the finger transitions which present particular problems. For me that turned out to be moving an index finger - its better mobility meant I was lifting it too high, delaying the time it took for me to return it to the instrument. What I then did was play pairs of notes using that finger motion - in my case, on a recorder, E-F and E-D, fairly slowly, and with dotted rhythms in both directions, so sometimes making the E the long note, sometimes the other one of the pair (this is on treble recorder: it'd be B and its neighbours on a C instrument). Thinking about the two halves of the process of making a note - lifting a finger, and dropping it down, as separate entities in this way will help you to focus on limiting your motion for a problem finger - and if they're all problematic, work on the worst ones first

. Looking at a couple of videos of my playing, I still do it, but can suppress the habit where I need to go fast, so although I've not fixed the problem completely, it can be improved this way.