You are cutting too deep. Just have the blade take the minimum off and make multiple passes to get the sidewall down. And buy one of these. By far the best and I have owned almost every style!! Vertically and horizontally adjustable and 2 different blades! View attachment 86789
A good sidewall skyver tool will have adjustability so that it can be setup correctly to only cut the right amount of material. Once adjusted to the ski, it's impossible to take too much sidewall because the hard metal edge stops the tool from cutting deeper.