It can depend on a lot of things. A basic one is the BRM guideline you've decided to use. Let's say it's 30 buyins. Any time you lose enough so you no longer have 30 buyins, it's time to leave. You can continue playing, but at a lower level.
As for other times, I don't use a fixed number. I play as long as I feel I'm +EV, which includes factoring in the opponents, my mood, etc. That said, I'd probably stop anyway if I lost 4 or 5 buyins.