There's no way to do it that wont annoy someone, so **** them. It's theire problem and your money.
Leaving should depend on the fish. If you just busted the only fish, wait an orbit and skip your blind to leave, that's what I usually do. If there's still a fish then you might as well stay.
At the end of the day if you feel uncomfortable, or you want to leave then just do it. Again, it's their problem, not yours.