In this blog post, we will explain how we handle security vulnerabilities in our software products, because it can be interesting for users to see. We believe that transparency is important, and it could help other organizations implement a similar process, or improve their existing processes.
We make software products, and we have a process to handle security vulnerabilities. The current process had its beginnings when we discovered some security issues in 2019, where we decided we'd like to inform our customers about them, and create a public Common Vulnerability Enumeration (CVE) entry for it. Those steps were more properly formalized into a process in 2021, and since then we've handled 14 CVEs according to this process (about 3-4 per year across 2 products, CFEngine and Mender).
We strongly believe that it's better to actively try to find and fix security vulnerabilities, and be open about them, informing our users and customers in a detailed and structured way. Security-conscious customers understand and appreciate this, and for others it can have an educational effect, showing the reality of software security and importance of regular updates for networked software, which could be exploited by bad actors.
For the purposes of this post, we won't go too much into detail on how to find security vulnerabilities, that's a huge topic in itself. But suffice it to say, we have several activities to help us find security vulnerabilities, including:
When a potential vulnerability has been discovered, disclosing it is the next step. For our internals / employees this can be done in different ways (creating a ticket, sending a Slack message), but for externals, such as bug bounty hunters, they need somewhere to contact to responsibly disclose the issue. We've chosen to use the standard security.txt format, to provide contact information and additional details around disclosing security vulnerabilities:
We've also set up redirects from other websites, so it should be easy for externals to find;
cfengine.com/.well-known/security.txt
In this file, externals can find instructions for how to contact us, which kinds of issues are likely to get a reward, etc.
Disclosures end up in the email list for the security team, where we respond that we've received the report and discuss it internally. We either create an internal ticket to handle the vulnerability, or we reject it.
When you have a program like this, with monetary rewards, you get various submissions which are not really highlighting exploitable security issues (sometimes dubbed "beg bounty submissions", often from anonymous gmail accounts). Thus, a good portion of the submissions are rejected, such as:
It's worth noting, even though you receive several low-value submissions, it's still been worth it, at least for us;
Based on this, it seems like it's been worth it to keep the submission as open as possible, to allow anyone to submit a potential issue, and then dealing with the bad submissions is a part of the job. The worst case scenario would be preventing someone from reporting a severe security issue which could damage us and our users and customers.
For the other submissions, which are accepted, we create an internal ticket, based on a template. The template has all the steps we need to perform (including some optional ones which only apply to externally reported issues, for example).
For the record, here is the overview of all the steps currently in the template, in our internal issue tracker:
Each step has additional instructions inside. We're not going to go through all of them here, but you might find the overview valuable or at least interesting.
At this point we need to understand and fix the issue. We create an internal ticket for the development team to fix the issue, and also do some other activities in parallel (such as requesting a CVE ID from MITRE). When fixing security vulnerabilities, it is a good idea to communicate closely and ensure that:
We communicate and work with our enterprise customers to give them a chance to upgrade or mitigate any security issues before they become publicly known. For each account we have at least one, preferably 2 or more security contacts to reach out to via email. The email contains similar information to the upcoming announcement, as well as a date when it will go public (normally in 1 month).
The public announcement is often identical, or very close to the email sent to security contacts. We typically use the same structure each time:
As an example, you can see our most recent CVE announcement here: CVE-2024-55959.
Once the announcement is published online, we submit the URL to MITRE so the CVE database is updated, we also add references to the CVE to our changelogs, so users can easily see which versions fixed which vulnerabilities.
You can see all the CVEs we've published for Mender and CFEngine on our blogs:
https://mender.io/blog/tag/cve
Since 2021, there's been 14 of them, which has meant that we can practice and improve our process over time. Here are some learnings to take away from this:
We are currently working with Hackerone, to start using their platform for bug bounties. With this move, we're hoping that the increased attention / talent and higher rewards than before, can help identify and fix previously undetected bugs and security issues. If there is interest, we can share some of our learnings on this later this year, or other aspects of security in further blog posts on the Northern.tech blog. If you're interested in reading more blog posts like this, follow us on Linkedin.