Prying-With a screwdriver, twist to raise the stem only slightly enough to insert the nail pulling part of the hammer (see below). Using the screwdriver only to remove the entire caster will result in your stabbing yourself and making a hole in your hand. It hurts!
Prying-This method may work. Use the nail puller end of a claw hammer. Place the nail pulling open end between the ridge on the castor stem and the chair base. Locate a leverage point to pry down onto and pull the hammer handle downward. Especially if you used a drop of oil, the stem should rise up a small amount providing an even better bite of the stem. When partly removed, you may need to place a thin piece of wood on top of the leverage point to press the hammer against for the stem to remove entirely.