Patty there are deterrents you can try but barn and cliff swallows are very persistent.
There are owl decoys, a natural enemy, just move them now & then so they seem more real. If theres a way to get a radio up there, pulley maybe, and play some hard rock obnoxious booming music all day.
Holographic scare tape, pie pans..... Some stuff looks like yuck but if it stops them its worth it.
When they swoop at you it is to get you away from their nest. The barn cats too.I have a stray kitty who comes up on the deck and they swop at him so much he leaves, figures the handouts at the restaurant are safer!
Once the nest is built tho dont take them dowm. A) Theyre federally protected unless its starling ior house sparrow
b) My neighbor took a cliff swallow nest off his house late last year and there were 4 babies in there that died.