Oh I've seen this kind of stuff in my garden before more times than I care to admit or remember, and since you are saying you aren't seeing the "bug" doing it, that tells me right there you have a night time pest... and for me last year that meant "Aphids" and damn it if I didn't struggle to see them and find them for weeks. So here is how I addressed it, I went out at night with a flashlight since I don't see it in the day I figured it has to be a night pest, when I shined the flashlight on the leaves I didn't see anything even in the night. Oh I was frustrated and I wanted to find it, thankfully I had another with me also looking and as they looked and I was searching, suddenly they saw something also green on the green leaf move which I kid you not blended right in... a couple of minutes of watching and then a black dot I thought was dirt popped out legs and begin to also move as well and then the more I watched the more I saw them both heading for the leaf's stem... and then I found out I was hit by aphids, a single one lays thousands of eggs in a week so by the time you even realize you have an issue... OMG BILLIONS if not TRILLIONS of them are your problem in under a month...
So I ran to the store bought the usual products for them Insecticidal Soap, etc... only to later learn a Aphids defense is to turn toxic chemicals in nature and otherwise into non-toxic chemicals they can pass through their body without harming them, do you have any idea of how much that becomes a problem when you try to wipe them out? In the end I was looking for all kinds of stuff, then I got hit with Mold again and I had the same product "Ed Rosenthal's - Zero Tolerance" pop up for both my aphids, and mold... and a friend was getting hit by spider mites and needed a product and again that same one came up as a solution. In the end I tried that product against all kinds of pests, including tent caterpillars and I was able to reclaim my garden to being pest free. The product was cheap enough for a gallon, 100% all natural, even qualifies for some Vegan thing, and unbelievably even smells like something sweet and warm in the kitchen I'd expect to come out of the oven. The only real issue or catch I had with the product was, they were kidding about trying it trying it on a portion of a plant to see if you need to weak it out, it contains essential oils such as cinnamon so it burned my plants really seriously when I first tried it until i was able to find the right dilution for it. But it did wipe out the spider mites, tent caterpillars, aphids, white flies and whatever those black spec dots with legs were. Grasshoppers even avoided my gardens and destroyed everyone's around me, so all in all I wasn't complaining and I think it cost me about $20 a gallon...
Just make sure you understand the best way to defeat a pest is by learning more about it, how it feeds, reproduces, etc... of all my books I have one "Marijuana Pest and Disease Control: How to Protect Your Plants and Win Back Your Garden by Ed Rosenthal" he gets into all the details of pests and why it all matters in a short but very informative way. First time ever I've not spend all summer long fighting pests week by week, and just had one big battle against pests and them kept them at bay the rest the summer in the garden of vegetables as well as other plants too. I seriously hope that helps you... because my personal guess if you have aphids yourself because that is exactly how my plants started looking... Later I found to go back and carefully look at the plant leaves and saw before the holes appeared it looked like something was eating cannons in the leaves for depth before the holes would appear... and thought it might be a mold not knowing that is how aphids work.