Det. Eng. Weekly #64 - xz: a wandering 🪿 event
An impromptu special edition about CVE-2024-3094
Welcome to Issue #64 of Detection Engineering Weekly!
Supply Chain Attack Vulnerability in popular “xz” library
Alright, I’m sure you’ve been blasted with news since Friday about the “xz” supply-chain vulnerability. If you haven’t, good on you. This week’s newsletter is a deep dive into the most valuable resources, commentary I’ve read about the vulnerability, and my musings on the topic. As security researchers and engineers, I think it’s crucial that you learn everything you can during these events so you can become better at responding to them in the future.
Emerging & Celebrity Vulnerabilities
In Vulnerability Land, it’s pretty rare to have an all-hands-on-deck, everyone-stop-and-patch event. The community sets the bar pretty high. In 2023 alone, 79.18 CVEs were released per day, with 36 CVEs in 2023 receiving a “perfect” 10.0 score. That’s 3 per month, but what makes the “xz” backdoor or other crazy vulnerabilities like Log4Shell, HeartBleed, EternalBlue, or ShellShock, well... special?
I helped build and manage an “emerging vulnerability” process in my last two positions. In this process, we treat classes of vulnerabilities like CVE-2024-3094 as security incidents—not that they are one, but rather that we run the same playbooks for ourselves and for customers just in case they turn into something crazy.
And what do I mean by crazy? You’d think CVSS 10.0 is enough of a reason, but since 3 CVSS 10.0 vulnerabilities are published monthly, I think it’s more than that. There’s a qualitative component to 3094, EternalBlue, and Log4Shell—a spontaneous confluence of exposure, exploitability, community outrage, news hype, and an “I can’t believe we let this happen” look and feel.
The “I can’t believe we let this happen” measure of 3094
As you’ll see in the links below, you’ll notice that 3094 isn’t an unintentional vulnerability or codepath, but a deliberate, intentional introduction of weaknesses into this library. So if a threat actor introduces a backdoor into code, is that a vulnerability? According to https://www.cve.org/ front page, it fits the definition:
CVE® Program Mission: Identify, define, and catalog publicly disclosed cybersecurity vulnerabilities.
And their glossary definition of Vulnerability:
An instance of one or more weaknesses in a Product that can be exploited, causing a negative impact to confidentiality, integrity, or availability; a set of conditions or behaviors that allows the violation of an explicit or implicit security policy.
So intentionality doesn’t matter, which means intentional supply-chain vulnerabilities are totally in scope. And guess what? It freakin’ worked.
Every security and infrastructure person had ONE term to refer to, instead of a stupid name. That term was easily indexable and searchable, regardless of product. Distro maintainers began pushing out patches and recommendations. Open-source tools picked these up and integrated them into their SBOM generation and comparison charts. Private vendor products had several checks around library versions and SSH versions.
As much as we like to shit on the vulnerability ecosystem, I don’t really know how much better we could’ve done here.
Still, like any “Emerging Vulnerability” I’ve taken apart of in the last 12 years, I can’t believe we let this happen.
What this means for Threat Detection
Since this disclosure, I've received many questions about how threat detection can help. A deep understanding of emerging vulnerabilities, especially their triggering mechanisms, can not only help stop threats but also help identify exposed systems.
In the reference posts below, you'll see that a very specific set of operating systems had the malicious vulnerable packages. In that scenario, you should only be worried if you were running an SSH server. That detection opportunity can be leveraged as a patching and scoping opportunity for your security program. Thankfully, the blast radius was much smaller than Log4Shell et al. That's where detection can be "more than just a rules team."
💎 xz vulnerability gems 💎
Subject: backdoor in upstream xz/liblzma leading to ssh server compromise by Andres Freund
The original source for most of the blogs listed earlier is that Freund discovered this vulnerability through benchmarking and finding strange behavior with ssh. He has a separate reply in lwn below that sums it up nicely:
Freund’s reply on a separate lwn thread:
A Microcosm of the interactions in Open Source projects by Rob Mensching
Burnout from maintaining open-source products is a real problem for the community, but do we have to add "a threat" there also? Mensching explores "Step 0" of CVE-2024-3094, where Jia Tang, the "actor" behind these malicious commits, basically gaslights the maintainer to take over the project years ago.
Mensching compares this to the malaise and stressful job of maintaining an open-source library full-time and that this experience is not unique from a mental health perspective.
Everything I Know About the XZ Backdoor by Evan Boehs
Boehs has the earliest and (imho) most comprehensive timeline of events leading to the malicious commits and the discovery on Friday. You can see the original mailing list messages that Mensching posted above.
Twitter thread by thegrugq on the HUMINT elements of the xz backdoor
I don’t want to steal @thegrugq’s thunder, but if you want an expert in HUMINT’s analysis on the subject, look no further than this Twitter thread.
xz/liblzma: Bash-stage Obfuscation Explained by Gynvael Coldwind
Excellent technical analysis of the obfuscation techniques used by Jia Tan to avoid analysis by folks within the open-source community. Basically, a multi-stage Bash obfuscation during build-time helped ship a malicious binary blob into the library, and unless you knew what you were looking for, you couldn’t really see it happening during the pull-request stages.
xz-vulnerable-honeypot by lockness-Ko
Now that you know how Tan backdoors the build process, you can see the payload and delivery mechanism on the other side with this honeypot. Absolutely incredible analysis of the shared libraries, and a clever way to patch the libraries with your own private key so you can "exploit" the bug live.
tl;dr Jia Tan injected code to hook how SSH verifies a signature on a fixed Ed448 private key, and if the signatures match, it passes a separate payload within the key to system() on Linux.