Here's the basic consideration in my mind: Do you think he would have called you with a Kx type hand? That is the typical problem if you have a player with a wide range of hands, they may be more likely to call reasonable pre-flop raises with paint and rag cards, hoping to hit something, doubly so for many players if the paint and rag are suited.
Now, maybe look at the numbers as well: 1/2 pot raise I would gather is maybe 4bb or so (you put 3, he put 3 as the bb, and the small folded). That isn't a really big bet really, considering what you put in if you call the odd there are only really 1 in 4 or worse, so there are plenty of hands that he could have and call with odds
Again, it depends on the player. This guy could be sitting with QJ and be more than willing on the flop to try to bet you out. He has 8 outs to make a straight, and another 6 outs to improve his hand (by pairing up) in that case, which may be powerful enough to merit the bet.
Perhaps it's my inner donk, but I would have called to see one more card in this case. If a Q comes, you have hit a house and you are pretty much golden against every hand he might have except KK. If the next street is a brick, then you can fold without feeling so bad about it.