RE: filesize...I typically stay under a GB with Handbrake (since I'm dealing with mobile video), and don't have a problem. How big was the one giving you trouble?
Btw, what is your conversion speed like? I have an older quad-core on Win7, and it tends convert in realtime (1:1 conversion time). Makes converting movies a slow affair, but fortunately there's a batch mode for big jobs.