Security audit of the OTR3 library

CoyIM uses its own implementation of the OTR protocol. This implementation is called OTR3. This library is written in Golang, and has since the beginning been extremely well tested. The goal of the library is to fully support version 3 of the OTR protocol, and also implement all features necessary to make it an easy library for clients to integrate and use. Since this library is the core of the most important security aspect of CoyIM - end-to-end encryption - we decided that we wanted to prioritize having an audit done, to ensure that it works as expected and doesn’t have any vulnerabilities or other issues.

In 2019-2020, we contracted the non-profit security firm Radically Open Security to help us do a full audit of the OTR3 library. You can read the final report here. This report was updated by Radically Open Security to include verifications of the fixes applied to the OTR3 code base. This audit reported six findings, all of which have been fixed in the current code base. Only one issue (OTR3-005) was remote, and could only be used to crash the process - not to extract any information. The fix of this problem was tracked in this issue. One issue (OTR3-004) identified missing file system controls. In this issue, the OTR3 library can export private key material to files that are not as restricted in permissions as they could be. This functionality is not used in CoyIM, since CoyIM stores key material in its own configuration file. This issue was easily fixed by applying more restrictive permissions.

Three of the remaining issues, (OTR3-001, OTR3-003 and OTR3-006) relates to key material being left in memory or not locked in memory. Since OTR3 is written in Golang, a memory safe language, that makes it hard to control what happens to key material in process memory. In a non-managed language, this would be easier to handle, but would open us up to a range of other types of vulnerabilities. Nevertheless, the three issues were resolved as much as possible, by using direct memory access to try to clean memory areas belonging to the standard library, and to use a library called memguard to protect memory which belongs to the OTR3 implementation. While it’s impossible to guarantee that these approaches will work 100% in all situations, they are sufficient to protect against these kinds of threats.

The final issue (OTR3-002) relates to the implementation of different algorithms that require modular exponentiation. The OTR3 library uses the standard library functionality to do these mathematical computations - something that Golang’s own cryptographic implementations also do. However, as described in an earlier section, this operation as implemented in the Golang standard library is not constant time. That means that it will take different amount of time to do the calculations depending on the private keys. If someone can observe these timings in some way (which in the setting of OTR would be non-trivial to do), they could reveal parts of the digits of the private key material. In theory, a remote attacker could learn the Hamming weight (the number of ones in the binary representation of the key) of an ephemeral key. Even though this was a highly theoretical possibility, and something that other Golang implementations of cryptographic algorithms doesn’t prioritize, we decided that fixing it was worth the effort. The way we solved the problem involved creating a new open source library called constbn, which provides specifically the big integer operations we need, implemented in a constant time manner. With this fix in place, OTR3 is not vulnerable to timing attacks.

The report makes it clear that the initial code had a good defensive attitude with limited attack surface, and that this attack surface was well defended. The conclusion says that “the code looks mature and has a good defensive stance”. Overall, resolving the findings above, added with the comments about code quality, the CoyIM development team is very happy about these results. We feel comfortable recommending that people use OTR3 for sensitive tasks.

We would also like to extend a huge thank you to Radically Open Security. It was a pleasure to work with them, and their support have made CoyIM and OTR3 better projects in the process. If you need help with security audits - especially for open source projects - we can easily recommend them.