The vulnerability is contained in libotr, short for OTR Messaging Library and Toolkit. The up-to-date version is now 4.1.1.
OTR stands for Off-the-Record Messaging. It's a a cryptographic protocol that scrambles messages sent through clients including Pidgin, ChatSecure and Adium.
The integer overflow flaw was found by Markus Vervier of the German company X41 D-Sec, which released an advisory.
"This flaw could potentially be exploited by a remote attacker to cause a heap buffer overflow and subsequently for arbitrary code to be executed on the user's machine," X41 D-Sec wrote.
The company found the flaw during a manual code review. It can be exploited by sending a very large message from one client to another, which causes an integer overflow that leads to a heap overflow on 64-bit architectures, X41 D-Sec wrote.
The message sent must be more than 5.5 GB. That's huge, but OTR allows for sending fragmented messages that are then assembled by libotr, the company wrote.
"Sending such a message to a Pidgin client took only a few minutes on a fast network connection without visible signs of any attack to a user," it wrote.
The latest version of libotr can be downloaded here.