Price: The most important consideration is to determine how much money you really want to spend on your new or used GPS system. You can spend anywhere from $99 - $1,000 so it is important to set your budget. The price will also determine if you are getting a basic GPS unit or one with all the bells and whistles! Garmin just came out with the Oregon, which has touch screen and allows for a wireless interface!
Portability or style: Do you want a hand held GPS or portable GPS? Or, do you want a car GPS or automotive GPS system that is secured in the windshield with a GPS mount? Portable GPS systems or hand held GPS devices can also be put in the windshield with a special windshield mount. Hand held GPS units are smaller and easier to carry when hiking, biking, etc…Whereas automotive or car GPS systems are designed more for travel and often speak to you!
Mapping Software: Will you need GPS maps or GPS mapping software? Do you need topo maps or topography? How about marine mapping? If you will be using it for boating, then you may need marine data. If you want driving directions than you need something that provides street data. These maps also usually provide restaurant data, tourist attraction, gas, food and lodging information as well. A GPS unit will usually come with only basic mapping data. All other mapping as I have mentioned here is extra and must be purchased separately unless you find a package deal.
Antenna: This is something else to consider. There are many different types of antenna configurations. Some are internal and some you may even connect an external antenna to all in the name of reaching as many of those satellites orbiting above! I mention this, just to be aware that there are different kinds and some seem to hold a lock better than others, especially in tree cover when hiking or geocaching.
Power: The power source of the GPS is important. If it is a car GPS or an automotive GPS, than it most likely can use a power adapter. However, if it is a hand held or portable GPS than you better have an extra supply of AA batteries on hand! Nothing is worse than being out in the middle of the woods and loose power! There are also power adapters for the hand held or portable GPS units that can be used in car to avoid running down your batteries.
Memory: One last thing to consider is the ability to extend memory for some GPS units with memory cards. Not all GPS units have this and they are usually found on the higher models. If you get one of these, I would suggest buying at least a 2 gig memory card to extend your mapping capabilities.