I use my t-shirts. Usually on the left sleeve, but sometimes on my stomach. Not a big deal to me at all. As everyone here has said, it's to be expected. Your skin produces oil and sweat, so its gonna get smudgy.
The ocd in me makes me do it pretty frequently, but I wouldn't say I overdo it. I like my fancy phone to look good!