Here are a few suggestions to remove caterpillars organically.
What you already did is easily the most organic method, just handpick them off. One is not a problem, one thousand is a problem. I find this a relaxing way to spend 10 minutes every few days or so.
Dust the plant
with plain flour. This bloats them and acts as a stomach poison.
Use Dipel dust.
Spray with pyrethrum and pestoil. Be aware that this may also kill bees and flies which are pollinating your cucumbers. It is a contact poison.
Also, be aware that caterpillars are most likely to eat the leaves so if there are only a few of them no big deal.
Encourage birds in your garden. Birds eat caterpillars, if you have a water supply nearby they are more likely to forage in your garden. Same goes for wasps, spiders and hoverflies.
You could cover the plant
with a mosquito net and this will keep all the bugs out. However you'll have to hand pollinate and that's more tedious than picking the bugs off.
Ensure your plants
have a good supply of potassium (potash) in the soil. This helps build cell walls and toughens the leaves making them unpalatable to insects. Potassium deficiency can be seen when the cucumber get a tapered shape like they are small at one end. This is most likely toward the end of the season.