The Chofetz Chaim reemphasizes very strongly how severe of a problem it is to speak badly about someone whether it is true or not true. To put down someone in front of others because he has a character flaw, whether he is not smart or strong or wealthy or any other flaw, even if it is true, if there is no positive gain in telling other’s about it constitutes lashon hara on a Torah level because since by speaking in this way you are hurting someone in some fashion, be it monetarily, physically, or emotionally by embarrassing him, making him depressed or even scaring him. For example if you say this person isn’t so rich then people might not be willing to invest with him or partner in a business with him. If you say someone is nit strong people might take advantage of him or bully him etc. etc.
The Chofetz Chaim says the worst insult one can give a person is to say he is not smart. To call a Rabbi, Rav, Rebbe, or Posek not smart is incredibly bad because not only might he lose his job and you are ruining him and his family if he is ran out if town, but you are degrading the honor of Torah because people won’t listen to him if he is trying to correct others who are not observing the Torah correctly since people will say why should we listen to him, he does not know what he is talking about. What will result is in people not observing the Torah correctly.