Not all dirt has harmful minerals; dirt composed entirely of organic matter (like peat) and inert solids (like perlite) is OK for CPs because it doesn't contain any minerals that can dissolve into water. True, perlite is a volcanic rock, which is a mineral, and peat is derived from organic processes that originated with minerals, but the point is that these aren't soluble mineral nutrients that a plant will try to take up and metabolise. When CPers refer to minerals in this context they're basically talking about stuff that acts as fertilizer like lime, potassium, phosphorous, etc., from what I gather. These free nutrients are harmful to the roots of most CPs and so are to be avoided.
~Joe