Ludwig said the vulnerability was initially reported through Atlassian’s bug bounty program on June 30th by Benny Jacob and that their security team quickly realized it was a critical issue. The patch was available by August 15, and security bulletins were sent out on August 25. They also submitted the vulnerability and patch to NIST and other government organizations so that it could be disseminated further. The information was sent out to Atlassian’s channel partners and account managers so that emails to customers could be sent out. Atlassian has its own test instances of Confluence and began seeing evidence of automated exploitation around September 1. Ludwig said it was bots probing the services and attempting to exploit them using the vulnerability. “As part of our normal process evaluating a vulnerability, we go back through the logs of our environment and our infrastructure and look to see whether there’s any historic exploitation. In this instance, we did not see any exploitation prior to our security advisory going out, but we did see it starting about September 1st,” Ludwig explained. “On September 3, having confirmed that, and also, having heard that there were plenty of folks that have not yet patched, we put out an update to our advisory saying that we have seen evidence of active exploitation and also encouraging people to patch.” Ludwig said Atlassian sent a second notification to customers after security companies and government agencies, like US Cybercom, began to send out notices about the problem. Despite Atlassian’s efforts, thousands of organizations were still vulnerable to the issue. Security company Censys found that the number of vulnerable Confluence instances was more than 8500 as of September 5. Jenkins, a leading open source automation server, announced on Saturday that its deprecated Confluence service was successfully attacked through the Confluence exploit. As of Wednesday evening, security company GreyNoise found that hundreds of organizations were still being targeted through the vulnerability despite the notices and news coverage of the problem. GreyNoise CEO Andrew Morris said there was a big uptick on Wednesday in Atlassian Confluence attacks, with “over a hundred devices opportunistically exploiting the vuln and counting. If you haven’t patched, you’re owned.” Morris told ZDNet that GreyNoise runs a large network of collector sensors in hundreds of data centers around the globe and saw the first opportunistic exploitation occur at 4:45 pm on August 31st. “We’ve seen it ramp up quite a bit in the last few days. And now, just today alone, we’ve seen over a hundred devices opportunistically attempting to exploit this vulnerability out on the internet,” Morris said, putting the number at 144. “All that means is that if Atlassian Confluence customers have not patched in the last week, it’s still extremely important for them to do so, but what’s even more important than that is probably calling an incident response team or network hunt team because there’s a really good chance – I would say like, 99.999% – that any Confluence customers that have not patched in the last week have probably been compromised.” Bad Packets reported that CVE-2021-26084 exploit activity was being detected from hosts based in Russia targeting their Atlassian Confluence honeypots. They previously said they “detected mass scanning and exploited activity from hosts in Brazil, China, Hong Kong, Nepal, Romania, Russia and the US targeting Atlassian Confluence servers vulnerable to remote code execution.” Of the instances in Atlassian’s environment, Ludwig said all of the attacks had been automated, and all of them have been cryptomining. Morris noted that it is hard to tell who is exploiting the vulnerability because many times, threat actors commoditize access, exploit new vulnerabilities, and then sell access to the system to other actors. “They could be some combination of APTs, criminal groups, financially motivated groups, government state actors, or even people that are trying to build up their botnet quite a bit. So it’s not altogether clear,” he said. “But usually when things like this happen, at least some amount of the bad guys are directly financially motivated, and usually the quickest path to monetization is using cryptojacking. In this case, I don’t have any evidence to suggest what the bad guys are doing once they compromised these devices.”
The problem with updates
Ludwig told ZDNet that the vulnerability is a “classic challenge that on-premise software has had to deal with forever.” “I remember 20 years ago, when I was at Adobe, we made a decision that we were going to start doing monthly security bulletins because that was a way to drive more consistency in terms of getting updates out there,” Ludwig said. “But even that level of consistency is just not sufficient to get people to patch on a regular basis. We’re fortunate that the Atlassian products don’t have, frankly, a lot of security advisories that go out. It can be months, if not a year, between when these go out. They’re relatively uncommon, but that also makes it a little bit more challenging to make sure that people are updating quickly because they’re not in practice the same way they might be for some of their other enterprise products.” He added that those who have internet-facing services and are not able to update in 24-48 hours should consider moving to the cloud. “You really need to consider getting to a point where your security is not dependent on the process that just doesn’t conform with modern expectations for how quickly you need to update. Right now, I don’t think we’re ever architecturally going to fix the fact that it’s hard to push out a software update, notify everybody, have them take action and do that faster than exploitation starts to happen,” Ludwig explained. Ludwig said Atlassian does not know how many organizations have not updated their systems or which ones may have run a script they provided as part of the advisory process for customers who did not want to update. Ludwig said he personally checked with customer support this week and noted that they are getting comments and questions as some run into issues updating their software. “In general, the volume of that has been lower than we’ve seen for previous security instances. So it seems like things are going pretty well,” Ludwig said. “For those who are attempting to do the update, it seems to be working. And the script also provides an easy way for people to make sure their environment is protected.” Ludwig added that they followed up with some customers on Friday and have provided Atlassian field teams with additional information. He told ZDNet it was difficult to know how many customers had been affected, how many customers are still not in a safe place, and how many customers are “not in a safe place because they’ve made a conscious decision.” “We will follow up when we can, but my expectation is that there will always be some number of instances of the software on the internet that’s out of date and that’s being exploited,” Ludwig explained. “Ultimately, we want to do everything that we can to make sure customers get patched or apply the scripts that they need to as quickly as possible.” A number of IT experts defended Atlassian’s response, saying it is typically difficult to get customers to update software, particularly during and after holiday weekends. David McNeely, CTO at ThycoticCentrify, said it was particularly difficult given that it simply takes time and, in many cases, requires changes to control approvals and subsequent downtime to perform updates or patching manually. Morris of GreyNoise similarly defended Atlassian’s response, noting that this kind of thing happens “pretty regularly.” “I think that when something like this happens, it’s really easy to rush and want to pile on to Atlassian for doing the wrong thing or making their customers vulnerable. They are responsible; I’m not absolving them of responsibility. But this happens to pretty much every software company on the planet,” Morris said. “From time to time, a vulnerability is disclosed, a patch is released, and then there’s a period of time where the vendor wants you to patch as soon as humanly possible. But they can’t make you do it.” This situation is particularly bad because of how many organizations are affected and because the timing – Labor Day weekend – was tough, Morris added. “It was kind of a perfect storm because Confluence runs on the internet, which means that it has to be resilient to attackers that would come in from anywhere on the entire Internet. It’s not like it’s buried deep inside someone’s network, where it would be a little bit safer by default,” Morris added. “If this is running in your environment, I would really, really strongly recommend patching and calling an incident response team.”
