Someone should make the SBF thread sticky for exactly this question.
$ent from a DeRezzed Droid2
Uhhhh... it
IS
Thurman: Everything you need to know about SBFing is in that thread. An SBF is the file that Motorola writes to the phone before it leaves the factory. It will overwrite every piece of factory software on the phone.
BEFORE you SBF, you should turn off your phone. When you turn it back on, hold the X key while you hold the power key. You'll get a triangle with an exclamation point inside it, and a little android. That's the stock recovery screen. Press the search (magnifying glass) key and use the arrow keys to select "Factory data wipe". This willl wipe ALL data that you have added to the phone since you got it. Your contacts and downloaded apps are linked to your google account, so they will be automatically re-synced later. However, if you have any important files, or save games or anything, you should back them up with one of the hundreds of free apps on the market. I personally use Titanium Backup, but it's not free.
After you do a factory data wipe, select "Wipe cache". Then reboot.
See if that resets it so that you can re-calibrate. If it doesn't, then you can try SBFing.
For the actual calibration, some have said to wave the phone over your head in a figure 8. I get the best results when I lay the phone flat on a surface and spin it 3 times in both directions, along all 3 axes. Make sure you're at least a few feet away from large hunks or metal, or electronics, as the magnetometer is pretty sensitive, and these can throw off your calibration.