As an addition, if you use the settings -> about phone > usage menu, that will tell you only the percentage of your consumed battery that different things account for, and it doesn't include an entry for GPS specifically. What it will contain is an entry for the app that is using GPS--you have to know that the app uses GPS. So it's kind of not useful if the app is GPS without you knowing it.
If you want to know truly what has used your GPS, you'll want to use the battery history statistics. Unfortunately, this is hidden deep in the sdk and there is no way to get to it through the settings. Luckily, there are apps out there that expose this screen to the user. Spare Parts (free) will show this screen as will the Battery Status and Battery Left Beta battery widgets. Lastly, bettercut (not free) will provide a shortcut to the screen.
The screen lets you see a lot of data about the phone such as how long the device has spent running (vs sleeping), how long wifi has been enabled, and how long wifi and bt have been running. It will also tell you what applications have stopped your phone from sleeping and for how long. It will also tell you what apps have used GPS and for how long.
I used to leave GPS unchecked, but now I leave it checked and just monitor that only the apps i want to use gps are the only ones using it. I tell my weather widget to use cell towers only and in general don't let apps use GPS if given the choice. I use that battery history page to see what apps try to use GPS, just to make sure that only ones i want using it are using it, and also to see how much time is spent using it since GPS is a big battery drain.
I found that ShopSavvy will periodically use GPS to update location for searches, regardless of if the app is actively being used or not. This prompted me to turn off location based searches and only turn it on when i actually open the app to use it. It's only like 59s it spends using the GPS, so not a battery hog, but it just annoyed me that it uses GPS when the app isn't even running.
So long story short, checking that box isn't turning on/off gps but rather giving apps permission to use the GPS. Like was already said, GPS is beig used when the satellite dish thingy is in the notification bar. If you want to monitor what apps use GPS and for how long, you can try using an app the exposes the battery history screen. It's very useful for seeing what is going on your device and using it to gauge the effect on battery life.