Hi marpez,
growing (or at least trying to) in your neck of the woods. My grow space is a doubled-stacked barrel that sits outside in the shade. My ambient temp/RH at the barrel location ranges from low to mostly upper 80's & 60-85%. My (passive) intake is 4" flex dryer hose, drawing from a room with thermo set to a range of 73-77f thru out the day. I have a 4", 190CFM inline fan on the exhaust port of my barrel. For the most part, I have observed temps that range from 75 to 82 with RH ranging from 47-60% over the course of the day. A big help is that my lighting(300w LED) is mounted on the lid with all the fans/vents for the lighting unit outside of the grow space. Created a solid negative air pressure using duct tape, flex seal, weather stripping and strategically placed plastic grocery bags. Pretty? Not so much, but seems to be keeping my temp/RH in check. Your closet probably has a bit more square footage than my barrel, and your lights are not vented outside of your grow space so perhaps a 6" fan with a higher CFM will be necessary to move enough air to keep things cool.
So, creating intake/exhaust ports for your closet and creating a negative air pressure might possibly be a way to get your grow temps to more closely replicate that of your thermostat settings of the room the closet is in. Good luck.