One of the biggest culprits in a Milestone/Droid feeling laggy is the way the stock Android UI works. Swipe from screen-to-screen and each screen needs to be reloaded and redrawn pretty much from scratch. The Nexus one has the same problem.
Get a home screen replacement like LauncherPro or ADW (or Sense UI for that matter) and notice the immediate "speed increase", a much more fluid feel to the UI, without changing anything else. Why is it faster? Because they're using RAM to cache the bitmaps, making for faster redraws. Stock Android doesn't do this, and that's where it feels laggy.
That caching is the real secret behind Sense phones feeling so much faster than Google Experience phones like the Droid/Milestone and Nexus One. It's not that HTC went through and fine-tuned Android, or that the Snapdragon is some Jesus processor. It's all about the caching. Which is also why they stuck a bit more RAM in the HTC Desire/Incredible compared to the N1, to make room for this cache.
So, get a memory manager app like MemFreeManager, set it to Agressive, use LauncherPro v0.6.2 (or higher) and set memory usage to High (to take real advantage of some of that freed-up RAM), and enjoy a much more responsive basic UI. Too many widgets or a Live wallpaper can still bog things down though.
As a side effect, having more free RAM means other apps will run better and smoother at all times, the downside being marginally longer load times for stuff that's been swapped out of memory more aggressively (keyboards are the most obvious things to suffer from this).
Good news is, this is offset by shorter load times for most non-resident apps because the memory that would have normally been swapped out starting when you launched it (and the overhead of clearing this RAM) was already done in advance.
Then you can overclock the hell out of it and
really fly.