I had a similar question about bees picking up pollen from poisonous plants. We have oleander, datura, poison sumac, and castor bean. Others, I'm sure, but those are the ones I know about.
Wherever the poison is in each of those plants, it is not in the pollen or nectar, so honey is no problem.