Is it offensive?

Your statement basically translates to "I only like your superficial traits" which is pretty bad. It's similar to saying you like someone just because of their looks.

I found myself having this conversation with my closest friend a couple of weeks ago and we both came to the conclusion that we are both superficial to varying degrees. It's one thing to not like an individual as a person, but a completely separate (and reasonably so), issue to not like someone because of their appearance. I know for instance that I couldn't be in a relationship with someone who is overweight. Not because I don't like them personality wise, but because their appearance just wouldn't appeal to me in that way. The same could be said of personality. I couldn't be in a relationship with someone who just drives me crazy all of the time because I couldn't wake up to that everyday.

And whenever I say this (especially to girls, omg) I get an earful over it. Is it so bad just to say, "I don't find you attractive"? Appearances are just another element of relationships to be considered.