You could have avoided that if you had used an ambidextrous safety that is retained by the sear pin. Brownell's sells ambidextrous safety kits that attach with the sear pin. Besides the fact that no stock panel modification is needed, an added benefit of a kit that attaches with the sear pin is that there is no wear on the frame, unlike those styles that use a tab under the right stock panel.
The ambidextrous safety kits Colt sold in the 1970s and 1980s used the sear pin attachment method. I have genuine Colt safety kits on several of my "shooter" Colts. I put a stainless sear pin style ambidextrous safety on my AMT Longslide, one of my favorite Model O-style pistols to shoot.