Wow - only thing I'd say for sure is that it wouldn't be an insta-fold for me. I might end up folding, depending on the circumstances, but I definitely wouldn't throw it away without any further thought.
It depends on all sorts of things:
Reads on the player? If he's a nit, OK this is probably a fold. If he's loose and aggressive, however, maybe we've got a decision to make. Has he done this hesitating thing before? What happened previously when he did it?
Cash game or tournament? If it's a tournament, what stage are we at? Are we near the bubble, or is his M getting low? Is our M getting low, for that matter?
How much has he shoved for? Does he have more money than us? If he's shoving whatever's left from 100BB, it might be hard to call if we're deep as well. If he's shoving whatever's left from 20BB... again, maybe we've got a decision. Or if we're short for whatever reason, maybe we don't have a lot of choice either.
Long story short, you need to carefully consider a range of things before making a call on this. Maybe it is a fold, but I wouldn't pitch the hand before thinking about all of the above, and maybe a few other things too.