The reason pot size doesn't matter is that you lose 3 times as much but you also win 3 times as much. When you factor that in, say it's 5 to 1 normally, so instead you win 15 and lose 3, that reduces down to 5 to 1 still. The basis is that even if a hand is for your entire stack, over the long run you will make that decision again several times, the odds will even out in the long run, thus for big pots and small pots with those odds things will even out.
Not sure if you're familiar with ev calculations, but basically they equate to how much you will win. Say a friend offers you a coin flip, if it comes heads they'll pay you $500. Your ev would be 0.5*500 which is odds of x happening and payout of x. If you sum all possible outcomes you will get the total ev. In this situation you are paid $250 basically. Sure it'll always be $0 or $500, but over the long run this decision to take it nets you $250. When playing poker we try to pick the decision that makes this number the highest. If we fold our ev is always 0 because we can win 0 chips but we also can't lose any chips. So if calling a bet is +ev, then we want to do it (unless we think a raise is more +ev, but most of the time draws like this it is not the case). So in your example, you're getting 5:1 on your money, so say the pot's 4, opponent bets 1 (so now pot is 5, costs you 1).
Say the draw is 25% to hit, or 3 to 1. 1/4 of the time you win 5, 3/4 of the time you lose 1. That's .25*5 + .75*(-1) = 5/4 - 3/4 = 2/4 = 1/2. So by making that call you win 1/2 of the bet. Say the pot was $12 and they bet $3, you win $1.50 every time you make that call. Sure you always either lose $3 or win $15, but on average you win $1.50 every time you make that call. Thus the pot size is irrelevant when making your decision.