I've always enjoyed hiking and heard about geocaching but did not have a GPS years ago. I finally bought a GPS and now it's something else to do while hiking. My favorite hides are in such a place I can park the car, put on the backpack, and head out for several hours hiking and finding caches on the way.
You can get basic models for around $100.00 up to $600.00 with all the bells and whistles. Newest one coming out had a camera built in.
I got the GPSMap 60CSx off ebay for $270.00 shipped to my door. When I got it the cheapest I could find online was around $310.00 shipped. Looking around online now I see they are around $275.00 shipped, so you might be able to find it cheaper on ebay.
It would probably be best to find someone who does this activity and tag along with them before spending money on one before you know if you will like it. For me, I would have a GPS for my hiking even if I did not geocache.