Sets vs flush and low draws in hi/lo

I've been playing quite a bit of 5 card Omaha hi lo, and I keep getting into situations where I have the ace 2 nut low draw with nut flush draw. However someone always seems to have a set against me and we end up going all in, with the usual outcome of my flush missing and making no low. When the roles are reversed and I have the set, the person with the flush or straight draw always seems to hit. How can i avoid this situation?