I'd agree Frank. The problem can be though that following a succession of seal changes, CVs etc etc that the shaft / seal might not have always sat in exactly the same place for all of its life leading to differing wear patterns. We know that there is end float in the whole set up and given that oil is held against the seal on the inside by a retaining ring, I think that diff oil gets sort of 'milked' through the seal when it moves back and forth. If the whole area is smooth then this doesn't happen but as you say, where there are grooves worn in the shaft, the oil gets through. But not, I'd suggest because the seal doesn't touch. I have put a seal back onto a worn shaft and it still touches even in the groove. Hence the idea that it gets pulled, sucked, milked or whatever via the wear groove into the swivel. Slight wear might not seem an issue but if as we suspect, that difference in diameter leads to even a small amount of oil bleeding through then using a sleeve is the right answer every time.
My issue is that no matter what I seem to do, my front diff oil turns to soup in only a few hundred miles. I've had the diff out, rebuilt it, cleaned the axle tubes thoroughly, replaced the inner oil seals, have no evidence of oil in the hubs and yet each time I come to inspect the oil, it awful. I've tried different oils, jacked the vehicle one side then the other to drain it, put oil through as a wash out and yet it's the same every time. I changed it before Russia, did 4000 miles, got back and it's the same. Breather is clear and raised and there's no leaks. It does this regardless of road miles or river crossings. If it was grease getting into the oil, it would only do this for a limited time until there was no grease against the inside of the seal. This has been going on for the last 5 years!
I don't get it.