Why would you be frustrated? This is a great situation. The players sitting out continue to post their blinds and you only have to play against 2 opponents. The other two players will likely be opening too tight, not calling wide enough, and generally not understand 3 handed play. The more players that sit out the better.