DON'T.
Colt frame pins are pressed in. Removal and replacement requires a press device with special contoured press-punches and support dies to press the pins out.
To remove you use a press-punch that's contoured to fit the head of the pin and a brass or hard plastic support die that is contoured to fit inside the frame and support the frame so it won't bulge or distort.
Installation requires a hollow press-punch that fits over the pin and guides and supports it so it can't go in crooked, and a brass or plastic support die to support the outside of the frame and prevent bulging.
These pins are "touchy" because do it wrong and you can stretch or distort the frame hole and the pin will be crooked or the hole will be stretched out of shape.
If you absolutely have to remove the pins either build the press, the press-punches, and the support dies or don't do it.
You're probably far better off to not remove the pins and just do what you can with them in place. You can do a excellent job with them in place by using Scotchbrite polishing pads from an automotive store.
DO NOT UNDER ANY CIRCUMSTANCES TRY TO DRIVE THESE PINS OUT WITH A HAMMER AND PUNCH.
Also, DO NOT attempt to remove the firing pin or bushing assembly. That's absolutely a factory ONLY job.
That assembly is also pressed in and out using an even more complicated press device and support dies. There's also a thin "skirt" of metal around the bushing in the frame that is easily damaged and you can ruin a frame easily unless you have the tooling and know how to use it.