I have been using perlite from various sources for years, unwashed to boot. I have never experienced any calamity due to flouride poisoning. Slowed growth? I can't say with confidence, but I never suspected my plants to be affected by anything.
I hear lots of folks say Plant A grows better in Substrate A than Substrate B, and that may be true in that grower's conditions. I always try to remember, though, that YMMV and lots of this type of information is mostly anecdotal.