What can I do about mushrooms growing in my ivy?

I have an indoor ivy plant which is in an appropriately sized pot and potted with Miracle Grow potting soil. I have had this plant for several years and it is beautiful and seemingly healthy. Other than the plant food in the soil, I do not use food or any additives, I just water it regularly. A few months ago, I noticed three small bright yellow mushrooms growing in the soil. I removed them and replaced as much of the soil as I could. Now the mushrooms are back! Am I doing something wrong? Are they harmful? How can I prevent them?