I have a pretty good 4G signal at work (Boston) and a moderate 4G signal at home (western suburb). I have found it makes a huge difference on battery drain if the phone is on 4G, and especially if it's not a strong 4G signal - even when the phone is on standby. Like others have said, I switch my phone to 3G mode when I'm not doing data intensive stuff, and always when it's on standby. For me, it makes the difference between the battery lasting only 6-10 hours (depending on use) when on full time 4G, vs. 12-16 hours of moderate use if on 3G most of the time.
Also, although I used to use the "LTE Switch" app to make the 3G to 4G toggle easier/faster, I had mixed results with it. As others have also noted, it seemed that sometimes I would loose data connection altogether when switching and would have to toggle back and forth again to get it back. Every once in a while would have to reboot to get data back. After about 5 days I ended up uninstalling it. I guess I'll have to put up with a few extra clicks to go into the phone settings to make the switch.