There is no point in getting annoyed about people making calling mistakes. Yes sure in this case you would rather, than he had folded. But he is going to make the same call against other players, and this will then benefit you, because you are the one increasing the value of your chips by folding and watching someone bust.
Instead you need to adjust to tendencies. If you knew, that BTN was calling to wide, would that have caused you to not jam ATo? Probably not because its a pretty strong hand from CO, and if he call to wide, he will also call with hands, you are dominating, like A9, A8 etc. But maybe you adjust by not jamming JTs, since that hand is rarely ahead of anything, that call.
Calling a shove is a math based decision, and because there are blinds and antes we dont always need 50% equity against the jamming range to make a profitable call. This is sometihng, that is often massively overlooked by people complaining about getting called by "bad hands". If the effective stack was short enough, calling with 44 on BTN can actually be a good and solid play.
This might be the core of your problem. If you play to tight, you are not winning enough blinds and antes, and you get blinded away, when the blinds go up. When you then finally has a premium hand, your main priority is to avoid getting drawn out on, which cause you to miss out of value. Because lets be honest now, people dont always "call anyway" regardles of your sizing.
And then at the end you are sitting with a very short stack like 5BB, you are forced to finally risk your chips, and you get called by 44, because pot odds are really good, when you are so short. Not saying that this is always exactly, what happen, but this is a very common pattern for people, who are playing to tight, and who dont know how to handle loose games, often because their postflop skills are kind of so-so.