Hate to break it to those labelling fish "higher lifeforms"... but fish aren't wired to feel pain. It's all just an involuntary stimulous response. They don't require fear and trauma to avoid damage... their brain simply tells their body to evade and it happens, without resistance. They and other simple lifeforms are basically organic robots, programmed with DNA.
Conscious beings like us need more than just a signal from the brain to stop ourselves from doing something stupid, and thus pain enters the evolutionary picture. And of course we start anthropomorphizing and assume that if we feel pain, everything must. But consciousness is a relatively recent development that requires a lot of hardware and a lot of energy to sustain it... most animals have gone on just fine for eons without it.
I think consciousness is implicit in the word "pain", which is what we get hung up on. That term doesn't apply to non-conscious creatures. There's just "nobody home" to experience it.
I'd never advocate injuring or killing a living creature for no good reason (it's very bad for our psyches too)... especially the conscious ones... but I'm just saying... if you're really heartbroken about the fish, you're the only one going through any torment here. The fish certainly doesn't care.