Rocking manual transmission while shifting??

It really isn't going to be anything more than annoying. the side to side movement doesn't move any gears around inside the transmission, only going up into the gears does. I do this sometimes but not between ever shift...i only do it when i stop before letting the clutch out to make sure it's out of gear. i'm sure over a very long time the bushings in the shifter linkage might wear out but if it's a shifter that goes directly into the trans the there really isn't anything to worry about.