Not entirely sure what you mean. But you do realise that the blinds could count towards most of it. 640 hands = 160bb* worth of blinds you are putting in before you've even had a chance to raise or fold. 20% VPIP is totally fine, but it's what you do postflop that matters the most.
How to win it bacK? Steal their blinds, stack donks, etc.
* you are putting in 1.5bb / orbit. 640 hands ~ 107 orbits. 107 x 1.5bb ~ 160bb. This is the best case scenario. We're not always 6 handed, so this number could be way bigger in reality.