It depends also on the whether you are referring to the latin or the common name. Some plants may carry a latin name that is also used as a common name for a totally unrelated plant. Agave is botanically correct for cactus, it may well be correct for lilys too.
Spec, spectabillis as I'm sure you know is a specie name which is commonly used more than once because names are actually desriptions in latin. However Agave is not a specie, it is a Genus and although I am not a taxonomist I do not know of any doubly used Genus names.
There are soooooo many poisonous cacti that most people don't know are. Just to name a few all the Trichocereus are hallucinogenic (poisonous) as well as Aztekium, and quite a few others.