Everyone has their own opinion on it. But it really depends on what you're going for: If you're going for the biggest, heaviest nugs, I would trim it off. But if you're really after the crystals for concentrates, then you may want to leave all your growth on, and use it all for trim when harvesting. People usually don't smoke the trim, but run it for concentrates. And if you let it all grow out, you will end up with A LOT of trim.
Personally, I would trim them off.
I'm an advocate for a lot of trimming. I Lolly-Pop my plants so much, other people think its a sacrilege. Really, my poor branches look like French poodles when I Lolly-Pop them.
By 3rd or 4th week of flowering, when they are done stretching, that's when I Lolly-Pop. I usually take off the bottom 50% of growth on each branch. Sometimes more, if the branch is big. But I'm sadistic like that.