Math is a huge field. Some of it is directly applicable to poker, but most of the probability math you'll use in poker is pretty basic  pot odds, etc. It's not something that takes a great deal of study to master.
As far as the more complex branches of math go, studying something like game theory could be useful, but I can't see a lot of the other branches being much direct help to your poker game.
I did advanced mathematics all the way through first year of university and honestly, if I had my time again, I'd go back and just do "dummies math" instead. That's the stuff I actually use every day, whereas I forgot everything I ever knew about the square root of negative one the moment I walked out of the exam...
