DEW #162 - Detonating TTPs with Agents, Writing Rules for Malicious Coding Agents & Skills Threat Models
Add a quarter to the jar everytime you say agent
Welcome to Issue #162 of Detection Engineering Weekly!
✍️ Musings from the life of Zack:
I had an excellent long weekend celebrating the 4th of July here in the U.S.! It was a good lead-up to a rather disappointing World Cup loss on Monday :(
We recently bought a kids’ WiFi landline phone thing so they can call family and chit-chat. Let me tell you: it’s been terrible. The quality/service is poor, and it just feels cheap. So, I’m trying my hand at rolling my own FreePBX server with an upstream trunk provider. I haven’t been this excited about a project in a long time :D will report back once I get it deployed
Detection & Response Happy Hour @ Black Hat
If you are going to be in Vegas during Black Hat, come swing by Tom’s Watch Bar @ the NYNY Casino right on the strip on Tuesday!
I’m running it back after BSides/SFRSA with friends and supporters of the newsletter, Cotool.ai. It was super chill at RSA with no vendor b.s., so escape Mandalay Bay and come talk shop with other practitioners.
💎 Detection Engineering Gem 💎
End-to-end detection validation using coding agents by Kyrre Wahl Kongsgård
This blog is one of the single best deep dives on detection validation I’ve seen in years. It hits a bunch of themes, including types of detection testing and architectural decisions for building and deploying end-to-end tests, and clearly describes an elegant and repeatable agentic loop for this use case. Let’s break it down piece by piece, because there’s a ton here and I highly recommend reading this one if you don’t want to touch any other story in this issue.
Security folks tend to steal concepts from SRE and developers and relabel them with fancier, cooler names, but the underlying principles remain the same. The picture above from Kongsgård shows how we celebrate concepts like testing and “chaos engineering” in security and map them to the security telemetry lifecycle. Regression tests, for example, focus only on verifying that an input (telemetry) produces an output (alert). Synthetic ingestion is an integration test of the ingestion and shaping of telemetry to generate an alert. End-to-end testing looks at the full telemetry → detection → response pipeline.
My favorite themes
Since this post has enough content to fill several posts, I’m going to point out two of my favorite themes so it doesn’t feel like I’m repeating or rewriting Kongsgård’s content.
Lab environment and TTP framework
Booting labs up to run simulations takes a ton of time and effort, especially if you are starting from scratch. The goal is to replicate your environment as closely as possible, but there are always trade-offs in simulation. Some of these tradeoffs include:
Environment mirroring: A host running in a VPC can help mirror what your endpoints or cloud resources look like, but it won’t be exact. Detonating potentially dangerous tooling inside a production environment can introduce externalities or even real security incidents if you aren’t careful
Baseline activity: A user, much like an agent, is non-deterministic. The telemetry they generate from normal activity is just as important to model as the malicious traffic itself
Provisioning discipline: Running a small amount of Atomic or Stratus Red Team tests is manageable from an individual detection engineer’s perspective. If you want to run your whole catalog of detections, you need to start thinking like an SRE or software engineer, as you’ll hit scaling and drift issues with your infrastructure
Kongsgård’s detonation environment has a high level of discipline to address these tradeoffs. The section on the Lab environment uses several DevOps paradigms, such as golden images, configuration management, and deployments via GitHub Action runners. Under the hood, they use Meta’s TTPForge as their adversary simulation framework for execution on detonation hosts. It offers a content-rich, multi-step attack-generation feature set that is adaptable to their agent-harness framework.
Agents as validation drivers and the schema knowledge base
Singular prompt, one-shot agents have their place in implementing agentic systems in security, but they tend to perform poorly as tasks become more heterogeneous. Since the task is end-to-end detection validation with TTP generation, rule tuning, and detonation, a Claude Code or Codex agent would not be sufficient.
The harness is the differentiator for anything agentic security. Remember that! Their agent isn’t improvising an attack; it’s following a plugin that teaches it to write TTPForge YAML files, ship them over SSH to a lab host, run the detonations, and then queries Splunk to see whether telemetry arrived and the detection matched. When a step fails, lifecycle hooks block progress until the agent finds the issue. Here’s the high-level architecture:
My recommendation to all my readers is to design your agentic workflows around single agents that do one thing very well. It’s as if you are extracting one piece of expertise from your brain and encoding it into a prompt to do a single thing. In this particular case, Kongsgård designed two plugins to perform discrete tasks.
detectionkit builds the TTP definition via TTPForge, writes the detonation test, deploys and runs the detonation. Its singular purpose is to replicate threat actor activity in a common & repeatable lexicon
splunk is the plugin that performs the validation that the correct telemetry was captured, the search for the rule was performed quickly, and continuously discovers index structure to understand rule performance and drift
Security vendors who sell agentic capabilities typically don’t expose their harnesses at the level of detail shown in this blog. Researchers and open-source enthusiasts are quickly catching up with these vendor-led harnesses, and this truly gives detection teams agency to choose between build and buy.
It’s been easier to admit that I can’t imagine a world without a Claude Code or Codex. The prompt was the star of the show for the first year or so of this coding agent frenzy, but it’s now squarely the quality of the harness that brings detection to the next level.
🔬 State of the Art
Detecting Agentic Threats in Claude: Writing Rules on the Execution Layer by Andrew Byford
This is a Part 2 post from Byford’s previously featured work on writing rules for Anthropic’s Compliance API. The cool part here is that, unlike his last post, which focused on the prompt content itself, this looks at the execution layer of the coding agents. I’ve always interpreted the execution layer as how the agent interacts with the filesystem itself. This presents unique detection challenges because, in my opinion, the impact is the same, such as downloading and executing a binary, but the paths are different, such as malicious skills, reading a malicious prompt, or loading a malicious plugin.
Byford splits the threat categories into five distinct buckets: excessive agency and permissions, supply chain threats, dangerous actions, sensitive information disclosure, and data poisoning. The architecture is clever where the Compliance API is used as an enrichment backdrop during investigations, so you can combine unstructured data from prompts with the structured data generated from Claude hooks:
And here’s the enrichment layer after a SIEM rule fires from the OTel collector:
Much like detection engineers have had to become supply chain security experts in the last two years, I don’t see a world where we also must become AI Coding agent experts in the next year or so. I never considered using prompt and response content generated from coding agents in the Compliance API as additional context for SIEM alerts, so I’m now going to steal that idea and see what I can do at my day job (sorry, Andrew!).
SOC Bench by DeepTempo
Evaluation datasets are critical for understanding model performance. Much like in my analysis of this week’s Gem, one-shot prompts can perform well under very constrained conditions, but without something to measure real-world malicious vs. real-world benign, you should limit your confidence in virtually all agentic security applications.
I found this SOCBench website & corresponding open-source repository, and it reminded me of Cotool’s Research benchmarks with similar datasets. This specific one includes a NetFlow dataset containing both malicious and benign network traffic. It’s also a bit more opinionated about persona benchmarks, ranging from SOC analyst to detection engineer, and includes more architecture, with tool catalogs and playbooks for those personas. Their first benchmark around detecting maliciousness:
Anthropic performed the best but it looks like it cost the most. I find it interesting that OpenAI’s benchmark had the threat analyst perform the best vs the SOC analyst in the other two.
Skills Registry Threat Models by Andrew Nesbitt
Two issues ago, I linked a blog by Aman Khurana that helped demystify the peculiar supply chain architecture behind VSCode extensions. The big takeaway I took from that blog is that not surprisingly, the more security engineers dig into supply chain security, the more they realize how difficult it is to piece together OSS ecosystems to perform effective detection and blocking. Coding agents are built to be autonomous and extensible, just like OSS. The difference lies in the non-deterministic way these agents perform coding tasks, due to intentionally designed boundaries.
In this post, Nesbitt unveils his threat model around coding agent skills. A skill is a bundle of prompts, code, dependencies, and tool permissions. Anytime a skill is used, the skill prompt is injected into the context window, and a set of tools and scripts gets exposed to the coding agent. The more frightening part of the Skills supply chain security is that, instead of a single npm command installing other packages in Node that eventually land a piece of malware, you can have a Skill install packages from virtually any ecosystem, and sometimes those packages are just more prompts.
I don’t think we are truly ready for a large-scale malicious Skill campaign, much like what we’ve seen with the likes of TeamPCP. Nesbitt points out several issues with how Skills are installed, deployed, and managed, and it certainly seems that this ecosystem is in the same stage that npm was in several years ago.
☣️ Threat Landscape
FBI Seizes NetNut Proxy Platform, Popa Botnet by Brian Krebs
The DoJ nabbed another residential proxy platform linked to the Popa Botnet. Krebs post here helps aggregate some of the data published by researchers at Google, Lumen and Spur. The wild part to me is that this proxy platform is linked to an Israeli company, and I’ve always assumed that these networks are owned by non-Western firms who are harder to work with outside of the U.S.’ sphere of influence.
These types of botnets finally figured out how to monetize without DDoSing. Krebs referenced research from Spur that nearly 50% of TV Apps on the LG Smart TV platform add the TV to these botnets, which are then sold as residential proxies.
I found a malware hiding in my tailwindcss config file. by Couch Potato
Super interesting write-up from a developer who encountered a Contagious Interview-style backdoor in their Tailwind configuration. They never figured out how it got there, but the indicators are classic Contagious Interview:
Targeting developers and backdooring their code
C2 server communication to an immutable blockchain style API
Rewriting git history to conceal the compromise
It didn’t necessarily say what the impact was or whether the campaign resulted in data exfiltration. They did find several unknown processes running in their production environment, so likely something happened there. If I had to guess, it was a PwnRequest due to the rewriting of the git history, but that’s about as far as I’ll go before I start placing bets.
Linux Backdoor Targeting iKuai Routers by dmpdump
This is a cool Linux backdoor writeup of a piece of malware that, based on my ~limited research, targets a Chinese-focused router typically deployed to East Asian/Chinese businesses. It’s an ELF binary that impersonates OpenWRT’s libjson_script.so.0. It was hard to ascertain at first, but it certainly is not a shared library and runs in userland. I don’t necessarily know whether the victimology is Chinese firms, which could make this a Western-based piece of malware, but it seems compact and very specifically designed for one router brand, which makes it smell like an APT implant.
ARToken: Inside an EvilTokens affiliate panel targeting Microsoft 365 by Michael Kelley
The TALOS research team uncovered an offshoot of EvilTokens, a device-code phishing-as-a-service kit. Kelley uncovers the initial BEC-style lure and then reverse-engineers the kit to find modern front-end components, such as single-page application lures, and a full backend dashboard written in React. Some of the differentiating features of ARToken Kelley found include keyword searching across victim mailboxes, post-exploitation tooling against victim SharePoint servers, and even collaborative session links for operators working on the same ARToken deployment server.
Lydia Graslie’s Gem from last week helps protect against some of these attacks, especially if you monitor which Microsoft management surfaces emit audit data for device‑code flows and token lifecycles, and treat gaps and schema shifts as first‑class detection problems.
🔗 Open Source
SOCBench’s open-source harness for evaluating alert datasets. The current dataset only contains NetFlow telemetry, but it looks like they want to add more. Their harness is the most interesting between playbooks, personas and how they run the evals themselves.
EasyTokens is a device code phishing toolset that emulates device code phishing as a service kit like EvilTokens. This one is more focused on performing the device code phishing attack itself, so you can use this to pivot into cloud and M365 environments.
Nox is an open-source attack surface scanning tool. There are 300 module plugins across 24 different categories. You can run each module individually or run a full scan that steps across all 24 categories to find everything from exposed credentials, vulnerabilities and OSINT findings.
phishdestroy/shortdot-evidence
Shortdot is a registry operator that hosts seven top-level zones (TLDs) that, according to PhishDestroy, almost exclusively contain fraud, phishing, and malware websites. This repository holds all of their research, enumerating the zones and their phishing website analysis across the seven sketchy-looking TLDs. Their research also includes financial research and how Shortdot charges ICANN fees to operate these zones.