The burr that is formed from diamond stones is so insignificant that I just leave it on, plus it adds some extra grab on hard snow for the first few runs. It might feel a little funny in soft snow but it takes a minute to remove if needed. I only file two or three times a season with 30 days on snow. Same deal, leave the burr, take it off if it's feeling funky on your feet, or just do a couple hockey stops on ice.
When I lived back East, I would purposely leave the (mostly horizontal) burr on, for the extra grip. I'd sometimes purposely built it up and fortify it with ceramic stones, for hardpan ice. (I seem to recall we used to call that a "bird's claw," out there gripping the ice.)
Out west, if there isn't any ice, I notice the burr - and thus usually take it off (1500 diamond file, as above). Also, I have had a few skis with a tendency to get hooky in particular conditions: on these I take the burr off as a matter of course. Not sure that helps other than psychologically, if the burr is slight.
Often, as mentioned, a slight burr wears off in a run or two. One can feel that too.