I think it is rare for people to just up and leave just because you choose to sit down. I play 2nl where people are not that observant but there is one weak regular I always try to play against if there are no good tables available. He pretty much leaves the table the second I sit down.
People leave for many reasons, mostly because they are tired, got stacked, or made enough money and want to bank their profits, or that the table has become nitty. Fish usually leave first in my view since they eventually get stacked or they leave with their profit, causing the VPIP of the table to die down. Since the fish are gone, the sharks leave for other tables and then, everyone is gone.