Outside of the world of open source software, few people have probably heard of XZ Utils, a small but widely used data compression tool in Linux systems. But late last week, security experts discovered a serious and deliberate flaw that could make networked Linux computers susceptible to malicious attacks.
The flaw has since been confirmed as a critical issue that could allow a knowledgeable hacker to gain control of vulnerable Linux systems. Because Linux is used around the world in email and web servers and application platforms, this vulnerability could have given the attacker silent access to vital information on computers around the world – possibly including the device you are currently using to read it.
Major software vulnerabilities like the SolarWinds hack and the Heartbleed bug are nothing new, but this one is very different.
The XZ Utils hacking attempt took advantage of the way open-source software development often works. Like many open source projects, XZ Utils is a crucial and widely used tool – and it is largely maintained by a single volunteer, working in his spare time. This system has created enormous benefits for the world in the form of free software, but it also comes with unique risks.
Open source and XZ Utils
First of all, a short refresher course on open source software. Most commercial software, such as the Windows operating system or the Instagram app, is “closed-source” – meaning that no one except the creators can read or modify the source code. With “open source” software, on the other hand, the source code is openly available and people are free to do whatever they want with it.
Open source software is very common, especially when it comes to the parts of software that consumers don’t see, and is extremely valuable. A recent study estimated the total value of open source software currently in use at $8.8 trillion.
Until about two years ago, the XZ Utils project was maintained by a developer named Lasse Collin. Around that time, an account named Jia Tan submitted an improvement to the software.
Read more: From botnet to malware: a guide to decoding cybersecurity buzzwords
Not long after, some previously unknown accounts popped up to report bugs and submit feature requests to Collin, putting pressure on him to hire a helper to maintain the project. Jia Tan was the logical candidate.
Over the next two years, Jia Tan became increasingly involved and, as we now know, introduced a carefully hidden weapon into the software’s source code.
The revised code secretly modifies another piece of software, a ubiquitous network security tool called OpenSSH, so that it passes malicious code to a target system. As a result, a specific intruder can execute any desired code on the target machine.
The latest version of XZ Utils, which contains the backdoor, would be included in popular Linux distributions and rolled out around the world. However, it was spotted just in time when a Microsoft engineer was investigating some minor memory irregularities on his system.
A quick response
What does this incident mean for open source software? Despite initial appearances, this does not mean that open source software is unsafe, unreliable or unreliable.
With all the code available for public examination, developers around the world can quickly begin analyzing the backdoor and the history of how it was implemented. These efforts can be documented, distributed and shared, and the specific malicious code fragments can be identified and removed.
An answer on this scale would not have been possible with closed-source software.
An attacker would have to take a slightly different approach to target a closed-source tool, for example by posing as an employee of the company for a long period of time and exploiting the weaknesses of the closed-source software production system ( such as bureaucracy, hierarchy, unclear reporting lines and poor knowledge sharing).
However, if they were to implement such a backdoor in proprietary software, there would be no chance of large-scale, distributed code auditing.
Lessons to be learned
This case is a valuable opportunity to learn about weaknesses and vulnerabilities of a different kind.
First, it demonstrates the ease with which online relationships between anonymous users and developers can become toxic. In fact, the attack was dependent on the normalization of these toxic interactions.
The social engineering portion of the attack appears to have used anonymous ‘sockpuppet’ accounts to guilt and emotionally coerce the lead maintainer over a period of years into accepting small, seemingly innocuous code additions, thus pressuring them to to cede development control to Jia Tan.
One user account complained:
You’re ignoring the many patches rotting away on this mailing list. At this point you are choking your repository.
When the developer claimed mental health issues, another account chided:
I’m sorry about your mental health issues, but it’s important to be aware of your own limits.
On their own, such comments may seem harmless, but in concert they become a mess.
@glyph@mastodon.social
We need to help developers and administrators better understand the human aspects of coding, and the social relationships that influence, support or dictate the way distributed code is produced. Much work remains to be done, especially to improve recognition of the importance of mental health.
A second lesson is the importance of recognizing “obfuscation,” a process often used by hackers to make software code and processes difficult to understand or reverse engineer. Many universities do not teach this as part of a standard software engineering course.
Third, some systems may still be running dangerous versions of XZ Utils. Many popular smart devices (such as refrigerators, wearables and home automation tools) run on Linux. These devices often reach an age where it is no longer financially feasible for their manufacturers to update their software – meaning they no longer receive patches for newly discovered security holes.
And finally, whoever is behind the attack – some have speculated that it could be a state actor – has had free access to a variety of codebases for a period of two years, conducting a careful and patient deception. Even now, that adversary will learn from how system administrators, Linux distribution manufacturers, and codebase maintainers respond to the attack.
Where to from here?
Code administrators around the world are now thinking about their vulnerabilities at a strategic and tactical level. It’s not just their code itself that they’ll be concerned about, but also their code distribution mechanisms and software assembly processes.
My colleague David Lacey, who heads the nonprofit cybersecurity organization IDCARE, often reminds me that the situation facing cybersecurity professionals is well captured in a statement from the IRA. In the aftermath of their failed bombing of the Brighton Grand Hotel in 1984, the terrorist organization chillingly claimed:
Today we were unlucky, but remember, we only need to be lucky once. You will always have to be lucky.
