Handbrake keeps crashing for some reason on mp4's havnt tried iphone mode. I use Visual Hub Its also free and from the same makers of handbrake. My settings are 854x480, a 1000kbps bitrate for video, and 128 for audio. It looks beautiful by the way, it also lets you up the resolution even though the source is less unlike handbrake. I also had alot of problems and spent alot of time working on it. Hopefully someone comes across this and see's what I'm talking about, video's are glitch free also. I'm using meridian for my video player on the droid. Here's the link to Visualhub since its hard to find since it got discontinued.
RapidShare: 1-CLICK Web hosting - Easy Filehosting