You either get vibrate mode, or you get silent mode. There's a toggle for it somewhere, but I can't remember where right now.
On the other hand, if you want both modes available, I use the "Silent Toggle Widget" that toggles from normal/vibrate/silent with a touch of the button.