DEW #164 - Your Benign Set Stinks, OpenAI Models Break into Hugging Face & Chinese Openweight Models
LIVE! CLICK! DRAMA! HACKS! RULES! CLICK CLICK CLICK
Welcome to Issue #164 of Detection Engineering Weekly!
✍️ Musings from the life of Zack:
I took a 7-hour road trip back home while the smoke from the Canadian fires still filled the air. It was creepy and somewhat pretty in a “holy shit am I on Mars?” way, especially when I didn’t need my sunglasses in the middle of the day.
The BlackHat Detection & Response Happy Hour hit capacity + waitlist in less than two weeks! Big shoutout to the co-sponsors, Cotool and Nebulock, for their support to help make this happen. I am going to try to bump up the capacity for the space even more, so register now if you can. I am also taking one more sponsor, so if you want to get your name in front of 100+ practitioners in the space, shoot me an email!
This Week’s Sponsor: Rilevera
Detections should be treated as code.
A rule misfiring in production is the most frustrating part of detection engineering. Attacker techniques shift, log sources change, and threat intel goes stale, you won’t know until it fails you mid-incident. Rilevera brings DevOps rigor to your detection program, from authoring through deployment.
Make your detection program more resilient with process-backed software and workflows. Rilevera gives you:
• Versioning & one-click rollback
• Peer review workflow
• Detection rule validation & efficacy reporting
• MITRE alignment mapping
💎 Detection Engineering Gem 💎
Your benign set should look malicious by Josh Waldrep
Detection & Response is the art and science of classification. I talk about labels a lot, especially when we start to consider rule efficacy as a function of a detection program’s success. It’s what SOC teams or incident responders immediately feel as pain. We may choose to incur the cost of a false positive for the sake of visibility, and in our heads we know it’s a good thing, but it’s still painful. This post by Waldrep helps explore labels while asking a critical question: “Are you being lazy with how you define false positives?”
The traditional AI or machine learning style labels include true positive (what we want), false positive (what we don’t want), true negative (what we want), and false negative (what we absolutely do not want). But, if you want to incur some cost for visibility, you may have heard the term “true positive benign”. It’s when you generate an alert where the traffic is benign, but you’ve accepted that you want to see it for a number of reasons, such as it being a critical asset or the rate is within acceptable parameters.
Waldrep’s argument, though, directly attacks the concept of a true positive benign and calls these types of rules lazy detectors. Using AI Agent detections as examples, you can see situations where a detection is lazy because the benign set you use to test isn’t using real data but rather synthetic data:
These are all fantastic examples of where, as Waldrep puts it, a detection did not have a realistic benign set for backtesting. It’s especially “lazy” because building benign sets is just as hard as building malicious sets because it’s rooted in your environmental context versus something you see online. For example, in the base64 image data URI scenario, you can alert on every base64 string you see. But if you aren’t feeding your backtesting set with images versus what is maliciously decoded base64, you may be testing on the wrong set of data altogether.
The ending quote is a pithy recap of everything in the blog post:
A detection earns its false-positive rate by staying calm when benign traffic looks like an attack. If it never met dangerous-looking benign traffic in testing, it hasn’t earned anything yet, no matter how good the number looks.
🔬 State of the Art
The Centralized Detection Model and Its Limits by Josh Kamdjou and Ian Thiel
I am happy to see this blog by Josh & Ian, who are friends of the newsletter, get published for folks to read! Full disclosure: I helped review the copy and theme for the Sublime team, but it is a great study of how email detection engineering is a different beast from pure SIEMs. This blog is Part 1 of 4, and it specifically addresses how email security vendors manage their detections under a “Centralized Detection Model”:
Most SIEM deployments you see combine vendor-provided rules, a.k.a. out-of-the-box rules, with custom or community rules. This makes sense in many ways: a threat actor who performs a malicious action within an environment, such as the cloud or an Active Directory domain, generates the same telemetry. You can replicate these attacks as closely as possible in your real environment and determine whether vendor or community rules reliably detect them.
The way I’ve interpreted this blog and series is that, as much as we think e-mail security is solved, which it looks like it isn’t, platforms that allow us to do modern threat detection have nascent capabilities to do so.
According to Josh and Ian, e-mail security vendors employ out-of-the-box rules just like SIEMs, but they tend to be more opaque. You can certainly replicate attacks over email, but if the underlying detection architecture doesn’t allow you to create, modify, or tune rules based on the environmental context of your organization, you risk becoming stuck while waiting for a vendor to update rules.
The team outlines three issues:
Explainability of ML & AI models is hard, and providing rudimentary configuration tools to adjust a threshold (0-100 scores) isn’t helpful. The more models you add, the harder it is to understand evaluations of detections, so when the only thing you can control is that damn slider to move it up and down, it’s not helpful
Suppressions and environmental context. Adding an IP address or a hostname to an allowlist is commonplace in detection. It’s brittle, but it does work on a per-rule or per-scenario basis. According to the team, many vendors provide a global allowlist, which can solve a false positive on one rule but create a false negative in the other
Emerging threats and relying on vendors to create or update rules. They call this “mean-time-to-coverage”, but basically, when a novel attack or vulnerability drops, you need to quickly build coverage. When you have to rely on an upstream vendor to ship a rule, you lose agency in the ownership of your rules
I’d be interested to hear how e-mail security vendors publish rules and provide these capabilities for detection engineers. Sublime does offer its ruleset as open source, and it has a community version you can integrate into your personal infrastructure. I promise this isn’t a shill; I’ve known the Sublime team for years and love their detection-engineering-first approach to a field older than most of us.
Stop Using OpenCode by Luke Wren
Ok, this is a very irreverent post complaining about OpenCode, but I promise once you get towards the end you’ll start laughing. I’ve been messing with open source coding harnesses lately, and Pi has been pretty great to work with. It’s very configurable and not really opinionated until you start adding a ton of plugins and configurations to it. This is by design: for example, if I want it to use subagents, I have to install a subagent plugin.
It’s split up into two sections, Annoying Things & Alarming Things. Annoying Things seem to be more rooted in harness engineering decisions themselves, such as how it manages memory and its context windowing. The “Alarming Things” section is where the security stuff starts and where I started cracking up. The wildest part to me is how permissions are persisted across sessions, whereas in Claude Code you’ll have a fresh slate every time you close and re-open the session.
I won’t spoil too much, but this quote from Wren is exactly the vibe sprinkled throughout the piece:
Clankers are not (usually) malicious but they are naturally adversarial because they are trained to compensate for stupidity with persistence. This is not a guardrail, it’s thoughts and prayers.
Kimi K3: Open Frontier Intelligence by Moonshot AI
The big news over the last week is the release of an open-source model, Kimi K3, that, according to their benchmarks, performs as well or better than leading AI Frontier Lab models. News like this is always crazy timing when labs like Anthropic are releasing their best models ever under Fable or OpenAI’s GPT-5.6 Sol. The Frontier Models are called Frontier because they a) perform the best and b) are clutched closely by the companies that train them due to fear of alignment.
Basically, if Mythos were open-source, would you trust those who can clone it into their Mac Mini to do the right thing hacking-wise? These labs generally say no. But when you see a release of a model that performs really well against the leading models, it makes you question whether or not that question is relevant because we’ll eventually democratize access to insanely performant models whether you pay for it or not.
They have several benchmarks, but the key graph is the cost versus score per task benchmark:
The economies of scale will likely protect OpenAI & Claude since they have moats other than their models, but this makes a compelling case for people to reduce cost for nearly the same performance.
☣️ Threat Landscape
Security incident disclosure — July 2026 by Hugging Face
Hugging Face, a very popular GitHub-style hosting provider for AI models, disclosed a breach from earlier this month. They claimed the breach was run, start to finish, by an autonomous AI agent, with no human operator confirmed at any stage. The attacker abused a vulnerability in Hugging Face’s data processing pipeline, where a malicious model managed to break out onto their Kubernetes infrastructure. From there, they harvested credentials and laterally moved throughout their environment. They claim there was no evidence of tampering with their or customers’ models.
The interesting tidbit here, though, is that as they were performing their forensic investigation, they couldn’t use Frontier Models for analysis due to usage and guardrail limits. Their security team had 10s of thousands of logs and commands to parse, and they were subsequently blocked from analyzing them with these Frontier Labs. Cleverly, the Hugging Face team hosted a local model which helped with the guardrail issues, and it cost them a lot less.
There’s nothing here about how they came to the conclusion that it was fully autonomous via an agent other than this:
The campaign was run by an autonomous agent framework (appearing to be built on an agentic security-research harness - used LLM still not known) executing many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services. This matches the “agentic attacker” scenario the industry has been forecasting.
OpenAI and Hugging Face partner to address security incident during model evaluation by OpenAI
OpenAI fills in the missing half of the Hugging Face story on what “autonomous agent framework” found and exploited the vulnerabilities outlined in Hugging Face’s security incident disclosure above. According to their post, the “autonomous agent framework” that popped Hugging Face was built by OpenAI running an internal evaluation harness.
They were testing GPT‑5.6 “Sol” plus a more powerful unreleased model on ExploitGym, a public security benchmark, with internet access and guardrails relaxed. The models realized the answers lived on Hugging Face, then chained vulnerabilities across OpenAI’s research environment and Hugging Face’s dataset‑processing pipeline to pull the solutions straight out of Hugging Face’s production database.
This is wild, right? It’s showing how determined models become when given a task or a goal they need to complete, and without the right guardrails in place, they can certainly complete that task with some crazy tradecraft. It certainly makes the case for the need of security alignment in light of the Kimi release news in the State of the Art section above.
🎙️ Podcasts
I’m back to listening to more podcasts, so I’ll post one or more episodes in a separate section. They will sometimes be new episodes, but I do think there is some great content that I’ll post that can be months to years old.
Pacific Rim is a threat group tracked by Sophos that specializes in finding exploits against routers, VPN and networking appliances. This specific episode follows several current and former Sophos employees who helped discover a Chinese-nexus APT that specifically targeted their Sophos Firewall product lines.
It’s a wild story because once they found out that the trial VM running the firewall was being used for developing exploits, they pushed an “implant” into APT’s firewall appliance to monitor them. They would download the threat actors' files, exploits, research, and even flood the network with ARP requests to fingerprint other victim appliances. It sounded to me like they were APTing the APT :).
🔗 Open Source
luckyPipewrench/agent-egress-bench
Pipelab’s eval benchmark for malicious agent egress attacks. It splits different attacks across 18 categories that you can use as a way to evaluate your internal detections. The key here is that this tests the tools catching these attacks from your agents, not the agents themselves.
manishrawat21/Cisa-KEV-Threat-Intel-Orchestrator
“Zero-touch” CISA KEV vulnerability-to-detection pipeline. It pipes vulnerabilities with known exploits into a structured prompt in Gemini to generate a Sigma rule. It tries to replicate the traffic with the correct Sysmon EventIDs and has several filtering mechanisms so it applies suppressions so the rule isn’t too noisy. Interestingly, it builds a log of discovered vulnerabilities to rule implemented as a compliance tracking mechanism inside a Google Sheet.
Furtex is a fairly comprehensive post-exploitation toolkit for Linux. It uses several techniques across two persistence techniques: io_uring and eBPF. It’s 100+ tools across four directories, and the interesting part here is its anti-EDR features. It targets Falco’s default ruleset directly, mapping all 25 rules to specific bypass tools.
The first technique is to stop the event from reaching Falco via clever io_uring manipulation, and the second lets the event through while breaking the rule condition via process name spoofing, path pivots, or alternate syscall flags. It’s cool to see anti-EDR techniques on Linux!
Grokpatrol is an open-source tool that checks if your Grok client uploaded your git repos and corresponding secrets to Google Cloud Storage infrastructure controlled by xAi. There was an excellent wire-level analysis on a Gist here, but tl;dr a background collector in the Grok CLI performed the upload outside the permission system of Grok itself. Grokpatrol scans several Grok artifacts for the “malicious” build that resulted in the DevOops upload.
Thanks for covering grokpatrol, looking forward to your feedback.
Obsessing over model outputs and detection noise is a necessary phase, but we must eventually move toward radical transparency in how those detections function. If we treat detection logic as a black box, we lose the ability to verify intent or adapt when the underlying model shifts. We need to treat the logic itself as the primary record of truth. When the code is readable, the entire security team can participate in the evolution of the defense rather than just consuming alerts. This transition from opaque triggers to clear, auditable logic is the only way to scale our operations effectively.
https://cyrilsimonnet.substack.com/p/a-detection-rule-you-cannot-read