Yeah, definitely remove the lower growth. It doesn't get enough light to make the good buds, but it drains the energy from the plant.
There is a whole technique called "Lollipopping", that's based on taking all the growth off the bottom.
This is a great example