Lots of different ways to go these days in terms of GPS devices. Some cell phones have them built in now, there are the auto versions (with mapping, generally quite a bit larger though), the general more hiking oriented devices, and now even a small device by Sony geared towards digicams.

I have a Garmin eTrex that I use to just mark locations. I then either enter the coordinates into my cell phone or I mark a waypoint and look up later.

The Sony GPS-CS1 holds about a full day of charge and then when done shooting, simply upload your photos, plug the GPS in and it will update all the photos with GPS coordinates (adding EXIF info to the actual files, similar to the way the camera records equipment and time on each file). Pretty slick device, just need to make sure the camera and the GPS are time synced.

When I take pictures, I usually set the bag down, break out the tripod, setup ... quite the production, so spending an extra minute with the GPS device isn't a big deal to me. So not sure I'd change anything at this point.