Oh gosh, a swarm of Goldfinches would be adorable, lol...
I have read that Goldfinches like both sunflower and thistle, I don't really see the harm of switching between the two, especially if you're attracting larger amounts. With some animals, generally tamed, you'd want to ween a new type of food in, but with wild animals they'll be less picky for a free meal, especially if it's something they like to eat. I think if you're attracting more it would be safe to say it's something that they'd prefer over the thistle seed
Have you thought of planting thistles or milkweed to attract them?