What kind of
plants are you putting in there?
I've seen things like philodendron and ivy grow beautifully in just water, with some pretty rocks or marbles...
I'm also wondering how it will look with soil showing through the glass.
BUT, if you're set on it, yes, I would put rocks in the bottom and be extremely careful about watering.
One thing I've done is to set small clay pots into the soil, with a cork in the drain hole, and add the water to that. The soil will take what it needs and the clay pot holds the rest. Not sure if you could find one small enough for a 4-inch pot, though.
This is a 6-inch pot with a 2.5/3-inch clay pot for water.