That is a harmless (to your
cactus) fungus (mushroom) that reproduces via microscopic spores. I am guessing that you are watering your
cactus more than necessary, but that is irrelevant... Wipe down the area around the
cactus and the pot with a 'wet wipe' that advertises that it kills germs etc. if it bothers you.
Personally, I have a yellow fungi living in my Begonia Rex's pot that I have been studying and nurturing...
just because it's a pretty plant (in my humble opinion)....