If you feel it causing frustrations between your friends then no. Most definitely a no if your friend is potentially a gambling addict. If you feel your friend has a habbit, which by what you say is likely then you should stop playing with him.
If you feel bad taking his money like you said, again you shoulden't be playing. Poker is about having fun and when you add money to it and you're playing with scared money like you along with the fact your friend has an issue with gambling then you should either completely drop the stakes or play for fun.
The last thing you want, is to be playing with scared money because it will put a strain on your friendship no doubt about that and it really isn't worth it, is it? If you fancy playing for stakes then go to a casino where you won't feel bad taking the money or upsetting someone.
The only time I'd play for real money
with friends and family is if the money was fun money and not likely to be emotionally effected by losing sums which you say makes you feel uncomfortable, this takes out all the fun.
Playing with scared money at the best of times vs unknowns is hard enough and bad enough, don't let it ruin things by doing it with friends and family, trust me - it isn't worth it from a financial point or view or friends point of view.
I did something similar when I was younger, except it was blackjack
and we was drunk. We all lost money to each other then it got out of hand where me and a friend did a "bet" whoever won the next blackjack won either of our cars. Girlfriends and friends around were not happy at the time, we shook hands the stupid deal was done. I won, he looked drained and ill. I didn't of course take the car but the emotionally strain at the time with alcohol inside us, I demanded the car and we fell out for a while. Anyway, he backed out basically and after sleeping it off and some weeks later we both realised we was being stupid and vowed never to play money games together. Thankfully we sorted things out and have never gambled together again.