Its a 2 step process:
Flush out the "My Contacts" group in gmail to include only those contacts that you want to see on the phone (Alternatively, I created a group called "Mobile Contacts" but see my edit below for the caveat).
On your phone, go to contacts, menu button, display options, press the gmail account, check on the group(s) containing the contacts you want to see on your phone. By default it should be "My Contacts"
Edit: When you add a contact from your phone, it automatically goes to the "My Contacts" group. If you don't have that checked on, new contacts will seem to disappear. I wanted to have things cleaned up and only include my Mobile Contacts group, but I'd have to add a new contact from my phone, then log into gmail and add that contact to the Mobile Contacts group. So, I'm just showing My Contacts on the phone.