I've been experimenting on what to do with clay, it sounds like you may have hardpan to me. Focus on short-term and long term goals. I'll also try to explain why. One of your short term goals is to stop adding sand. Like mentioned you're taking a big risk, unless the sand is very sharp, mixing the two can create a cementlike ground that resists root growth along with preventing air & water penetration and what you'll have is something worse than you've got.
Here's your short term goal. Incorporate organic matter probably far more than you can imagine and till it in, deep. Make sure the dirt isn't wet when you do it. The reason to avoid doing it while wet, is that you'll destroy what soil structure you have (if wet the clay particles seep down and the sand particles stay at the surface screwing up the structure). You want to till deep because you'll be creating a "boundary" which is pretty much a difference in the soil. Boundaries are not good, and will be part of your long term goal to remove them. In this case, if you till 8" deep then the boundary is going to be 8" down between the soil you tilled and the soil you didn't. Roots don't like to penetrate boundaries and part of your long term goal to fix it.
Why organic matter? Organic matter coats clay particles physically seperating the particles and aggregates from each other so it acts more like normal soil. More importantly micro-organisms that eat organic matter produce glomalin that bind individual clay particles to each other which reduces crusting, water infiltration, and reduces erosion.
Why so much organic matter (compost) is that most of the compost will break down in short order, but there's a fraction that doesn't and that fraction is the part of your long term goal. The bad is, tilling the soil hastens the decay of the organic matter and keeps that boundry active. However, we need to get it in there and started. So another part of our long term goal is to reduce how often we till to get to a point we no longer need to, and it can become self sustaining and won't require so much organic matter.
So, here's the process.
Short term. Get your soil tested and figure out how much lime & phosphorus you'll need. You're best to till it in instead of let it rest on the surface. Next, cover the area with 3-4" of compost or apply layers of compost, till, and apply another layer and till it again. That ensures better mixing but more work. You should wait at least 3 weeks then
plant whatever. In late summer or fall, cut everything to the ground and
plant red clover, which has huge strong & deep tap root and should aid in breaking the boundry you created while adding nitrogen. Come
spring, chop it down and put more compost on top of the clover and till it into the soil. Wait 3 weeks, and
plant your garden. In fall check your soil, if your soil is looking good you want to phase out of the red clover and move over to a cover crop each fall, and in
spring chop it all down to the dirt and don't till it in.
plant your garden, or let your perenniels grow through it.
You should see a small improvement each and every year. You need to maintain either compost or a fall cover crop or both not for fertilizer but simply to maintain/improve the structure of your soil.