I don't think so.
Worm castings = fertilizer. Add to soil for plant food.
Perlite = small rocks to help with soil drainage. Soil with fine to super fine minerals have very poor drainage so by adding perlite you now have a mix between medium, small, fine and super fine. Which allows for better drainage because there is more space between the particles.