Is it insulting?

I've always been called "hard-working" at work, but never been called smart. In fact, a professor actually said jokingly that I tend to not use my brains. And implied that I'm not very bright.
I wonder then if being hardworking makes it a joke if I'm not smart enough.