Have you ever gone out to your garden and notice what looks like sand soil dug out and wondered what was living in there? Then you notice it, the airplane of a bee hovering around the hole. Holy smokes! It's a flying cow. These bees that come out around this time are called cicada killers. A very large wasp that feed on nectar, but their young, the larvae, feed on cicada.
So what do you do? Well these wasps don't really hurt you if you leave them alone, but depending on where the holes are situated, you may need to eliminate them. I think the best approach is either to give a broad spray over the wasp and the holes, or dusting directly into the hole. Just use caution, and if you are nervous, call a pro.