Like Jeeper; I've clipped it upon the sternum strap if I felt I was going to be need it readily.
On rare occasion I keep it around the next, but that's best for level trails where it's not bouncing against me too much.
Honestly, I often have it stored in the upper pouch of the pack. While not readily accessible, (you either drop pack or have a hiking partner grab it out) it has worked fine. Perhaps because I've seen my share of moose so that I tend not to grab for the camera anymore and instead just enjoy the moment, it works out fine. The one wolf I have seen was so quick, I don't think I could have got a shot had I been walking while looking through the viewfinder..
I often do find that I don't take many pictures while actually "on" the trail, hiking. For one part, it's a great excuse to drop and pack and have a break. For another, I'm typically on the trail while the sun is already fairly overhead. The amateur photog in me prefers the early morning and evening light for those great photos..
Tom