Kabbalastic drashot

The rabbi in my local schul is fond of bringing kabbalah in his drashot, however, I know that one who is not a talmid chacham is not supposed to learn kabbalah. Would it be better, al pi derech eretz, to leave during the drasha or to stay in the schul.