Wittners are a slightly more difficult install than you might think. Having done this recently here are a few pointers I did not find anywhere else:
First, the position you are going for is very important:
A big thing is the width of the moving part is less wide than the peg box, so where to position them is critical, wrong positioning and they won't work because strings will break if they wrap onto the non-moving part.
The G and E string moving parts need to be matched up to their pegbox respective sides, with no fixed (non-moving) part exposed, or the string will catch and break.
This is because the string angle for those two strings forces the strings right up against the edge of the peg box.
The two middle strings can be closer to center, but should still be positioned to expose less fixed-shaft on the side where the string is.
Next, is how to avoid overreaming the hole. This is important, if you blow this, the easy fix is to order the next size up of Wittners and try again, but once you top out you are stuck either gluing or going to a luthier for expensive peg box repair.
It is also not possible to do one string at a time, you will have to remove both pegs on a side for enough clearance to use a hand reamer, or find one with a longer handle than mine.
I used this procedure (repeat until done): - ream a bit, the closer to finish, the smaller the increment should be. - test the peg, press in, not too hard, but not too gently. - to remove the peg, use pliers and a very thick towel to avoid scratches, this takes some skill, I had no troubles, gentle pressure and it pops out. Removing a Wittner by hand will rip the knob cover off, exposing gears and grease. Be careful with the pliers to not squeeze too hard, and get the angles right to not stress the peg box too much. Destroying a peg box is not what you want to do. If you don't have a gentle hand for this kind of work I would recommend going to a pro. I"m a wood worker and amateur guitar tech for a long time so this wasn't a huge stretch for me, but I did have to go the next size up and retry on the first violin I did.
Ream until almost there, then do the last bit by pressing a bit harder, don't overpress, but you don't want them loose either, or they will slip and it's either larger Wittners, glue, or pegbox repair.
I did the first peg hole with the reamer, in careful increments, just as you did. When I got it to the "just right" spot, I marked the relief edge of the reamer with a "Sharpie" pen where it hit the outside of the pegbox. This gave me an indicator of when to slow down/stop, which has worked fairly well on the 3 or 4 sets I've installed so far. An alternative means of removing the pegs during the test fitting is to get a small dowel, slightly smaller diameter than the end of the peg shaft, and push/tap the peg until it loosens.
To do this, you put in the peg, make a mark a few mm from the box so that it would still stick out a little if you only cut them off. Then with coarse sandpaper or better a sandpaper sponge you roll the end around in a circular motion then continue with finer grades until its basically just polishing paper.
I have a very rough fiddle that I put in conventional, modern taper (30:1?) pegs but left them long. That extra bit sometimes interferes with my fore-finger when playing, say, an E# on the e string up by the nut or a Bb on the A string. When it comes up it's a nuisance. Otherwise, I don't care. I still haven't shortened them. Eventually I will.