In the link below you find a benchmark comparison for the OMAP 4430 vs the Tegra 2.
AnandTech - Motorola Droid 3 Review - Third Time's a Charm
In nearly every instance the 4430 is faster than the tegra 2.
Keep in mind the 4460 is clocked much higher than the 4430 so the scores would be even better than what you see here although the higher resolution of the nexus will likely take some hit on performance it should still perform nicely.
Is it as fast as the iPhone 4S no. Is it more than sufficient? YES!
This is just my two cents. I don't do much gaming on my phone so maybe if your a gamer you have got more to worry about but for everybody else stop worrying!