DEW #167 - More Security Agentic Breaches, Cheap and Scalable Threat Hunting & Detection Skills
"We want them to practice hacking stuff", "nooo not like that"
Welcome to Issue #167 of Detection Engineering Weekly!
✍️ Musings from the life of Zack:
I had a fantastic time at Hacker Summer Camp! Thank you to everyone who came to my Happy Hour, where we had nearly 300 registrants!
I did an awesome panel with Jack Naglieri and Damien Lewke on SIEM and security operations hosted by Decibel Partners. Shoutout to Dan Nguyen-Huu for wrangling us together and giving me the creative space to try something a little different with the format
Last but not least, I hosted a Detection Engineering panel at the Blue Team Village with leading experts in the space: Christina Parry, Chris Kulakowski, Alex Hurtado and Chase Phelps. According to the organizers of the Village, it was one of the largest turnouts they’ve ever had. The line to get into the talk stretched well beyond the village.
DEFCON badge plus a 40-oz malt liquor badge. Yes, I was that guy in college :D
Sponsor:
Your security architecture matters more than your AI model
Security vendors are racing to ship better AI models, but their architectures are dragging down their models. Even peak AI can’t overcome a limitation like a global detection model shared by all customers.
Join Josh Kamdjou (CEO, Sublime) and Dmitri Alperovitch (Co-Founder, CrowdStrike) on September 10 to hear why detection architecture determines what defenders can actually do to close gaps, and find out what it looks like when your platform can autonomously close gaps in hours, not on a lengthy vendor update cadence.
💎 Detection Engineering Gem 💎
Every time I attend Hacker Summer Camp, I try to survey the thousands of talks and build a rough logistical plan on how I can attend them. The plan never works because Vegas is way harder to navigate between the casinos, taxi lines and long Uber waits, but if I can attend 20% then I know I’ve done well. So when I saw this talk on the HuggingFace incident by OpenAI security on the schedule, I got excited because it was featured on my newsletter and it spawned subsequent disclosures by other labs.
To the surprise of no one, I was too far away at an event to attend this specific one, so luckily Black Hat published the video :). I highly recommend watching or at least listening to the briefing. The security incident details are, IMHO, insignificant in the sense that they are nothing unique to what we haven’t seen before. The details behind the evaluation environment is the most fascinating.
We know that the Foundational Labs have model evaluation environments to help determine the efficacy of their newest model releases. Of those evaluations, OpenAI and many others run ExploitGym as a cybersecurity benchmark. OpenAI’s ExploitGym environment did not have Internet access to make sure it couldn’t cheat, but it could use internal resources and tooling, such as Artifactory listed below.
The OpenAI team runs thousands of reinforcement learning evaluations on these models, so they give what they call “impossible tasks” to agents to see how they perform. These tasks can include writing an exploit for a vulnerability, or performing some calculation in a Google Sheet that requires Internet access. This helps guarantee that the models don’t “cheat” by using the Internet and can perform the task at hand. The impossible task runs across May yielded three distinct things:
Agent runs found file write access (I’m unsure if this is a vulnerability, it was hard to tell in the presentation) to Artifactory which created a pseudo-message board scenario. Agents used this message board to communicate with each other
An agent found an SSRF in Artifactory itself, which gave it internet access to help solve the impossible tasks
A separate agent found an RCE in Artifactory from a legacy auth endpoint, giving it command and control access to Artifactory
These three things are important, because OAI found this activity prior to the Hugging Face Incident. They patched the vulnerabilities and redeployed the infrastructure. The issue was that this data was held in the memory of subsequent agent runs because the data made it back into normal agent evaluations. So they had a memory that this was possible!
The agents used this exact memory to re-create the message board and the subsequent attack paths through Artifactory into Hugging Face. They found different vulnerabilities and were so obsessed with their “impossible tasks” that it resulted in a real world breach. The agents determined that Hugging Face contained the shortest path answer to completing their evaluations, so they found a deserialization vulnerability in how Hugging Face loads dataset evaluations and chained that with a Jinja injection vulnerability to gain a foothold into Hugging Face’s environment.
I highly recommend watching this, it was a trip and one of my favorite talks of the conference.
🔬 State of the Art
Threat hunt AI: How we built an AI security analyst on AWS for under $500/month by Pablo Vidal
This post by Vidal outlines how their detection & response team runs their autonomous hunting and triage agent. It’s built on Serverless architecture on AWS: agents run as a container on AWS’ Elastic Container Service Fargate, and the agent loads in several hunting and detection configurations via YAML. It uses Snowflake as their longer term storage for alert and telemetry data while using 2 weeks of log, metric and traces data on Datadog (yay!). It integrates directly with their alerting and investigation workflow via JIRA and Slack.
The interesting design choice here is the multi-phase investigation. Sonnet is used as their data gathering agent, while Opus compares potentially malicious behavior to one- and seven-day baselines. It’ll enrich suspicious actors and resources, scores confidence, then uses a separate Opus agent to perform adversarial validation.
The final adversarial phase must produce specific counter-evidence before it can dismiss a finding, such as an allowlisted service account or an established false-positive pattern. Vidal says this reduced noise from roughly 40% of findings to a level where most results are worth analyst review. The title does say it ran for under $500/month, but that is an additive spend since we don’t know what their Datadog & Snowflake spend is.
Introducing Deputy: Better signal and control for software supply chains by Kent Gruber
Deputy is a rule-based language for detecting and enforcing security controls on vulnerabilities found in CI/CD. I included this blog in particular due to its similarities with how security operations deals with threats. The tool introduced in the blog, Deputy, helps find and alert on potential policy violations when code is being introduced into a source code pipeline. It uses a YAML based language, and the rule language is CEL, which we’ve featured on a previous issue with Perplexity’s numbat project.
There are nearly 60 policies in the repository, and each policy has several rules each that help enforce the policy. Enrichment is a key component to the policy engine, Deputy precomputes values to pass along to CEL so the rule language can alert on things like critical vulnerabilities, the presence of a vulnerability in KEV or a new container image that contains a critical vulnerability.
According to Gruber, the project is early and it will not replace reachability or application-context analysis. But the policy and rule model is worth studying. Supply-chain controls are more likely to survive when the same decision logic follows developers, CI, package proxies, and coding agents, and it introduces the funnel of fidelity effect on an attack surface not traditionally monitored by security operations engineers.
Detection Skills by Vega Threat Research
Detection-as-code has gained a ton of popularity in the last few years because it helped build resiliency in the quality of a rule as it moved through ideation, implementation, review and deployment. These pipelines help do everything from linting, to applying governance checks around reviewers and deploying only via Terraform, as well as applied testing via Atomic Red Team and the myriad of other testing suites. Agent skills are the next logical step of adding security agent steering-as-code to triage, response and detection engineering agents in prod.
I ran into Eli Rozen at the Vega booth at Black Hat, and he showed me their research around Detection Skills. This is a great concept: as detection engineers, we need to think about codifying the investigative and analytic knowledge we use for alert triage and tuning as much as the detection logic we deploy everyday.
The Detection Skills OpenSpec is Vega’s contribution to the detection engineering world that helps keep the detection logic in the detection platform, then attach a portable SKILL.md package containing the triage, investigation, and tuning work surrounding an alert.
A detection can start with a deterministic query, invoke an agentic triage skill, pass its result to an investigation skill, then send the outcome to a tuning skill that proposes a change for human review. According to this spec, skills can bring focused references, lookup tables, prebuilt queries, and enrichment scripts without loading every detail into an agent’s context window upfront.
The Vega Threat Research team published 50 skills with this initial release, so it’s really cool to see them build in the open and share their knowledge with everyone else.
ATEN: Endpoint Telemetry for AI Coding Agents by Anton Ovrutsky
Over the last two weeks, I’ve featured “AI-EDR”-like releases from Perplexity’s Numbat & Uber’s ADR projects. The development of security telemetry on coding agents makes a lot of sense. They have agent in the name, so much like an EDR agent on a host or a CNAPP agent running in the cloud, you can extend Claude-et-al to generate contextual coding session telemetry. This is the first post I’ve read on the concept of AI-EDR that brings receipts of how it works during an investigation.
In this post, Ovrutsky releases his ATEN project which provides coding agent security telemetry. Unlike Numbat & ADR, it’s observe only, but there are some unique elements to it that differ from those two projects:
It reads each prompt and creates an intent trajectory, meaning it’ll enrich the prompt with additional information such as tool calls, agent session and results. This helps create an index of sessions to answer the who later on
It uses Kernel-level collectors (Windows ETW, Linux eBPF) that records the action taken on behalf of the above intent. These collectors focus on host and network telemetry
It combines the intent and the Kernel telemetry into a full picture, reconciling the underlying action of the prompt and tool call with the on-system activity
He has several examples of running prompts and viewing the output in Splunk. The Splunk queries and subsequent responses expose the intent and action from the session and gives a much richer view into what happened.
For example, he issued a prompt to run a health check inside a code repository, and the health check was a PS1 script that ran additional Claude prompts. The process chain is a mess if you look at EDR or ETW logs, but with ATEN, you can follow the chain-of-thought via the intent and the kernel actions in one view.
☣️ Threat Landscape
Lol agentic cyber breaches and the hype train
I am trying not to be facetious here, but I do want to point out some news announcements of organizations besides OAI/Anthropic who are disclosing security incident breaches due to training models with no cyber guardrails. I think it’s important we keep reading about these for two reasons:
It demonstrates that we need to seriously consider an agentic or autonomous approach to defense, as we will likely expect more of this in the future
The absolute craziness that these incidents come across more as cool than “dude you just breached my company”
Incident Report: unsanctioned agent behaviour during cyber testing by AI Security Institute
The AI Security Institute was running cyber evals with Mythos 5 and GPT-5.6-Sol with guardrails removed and found that less than 10% of their runs contained evidence of models trying to solve a challenge by attempting to breach organizations outside of their institute. The crazy finding here, IMHO, is that one of the models tried to insert malicious code into an open-source project and engaged in social engineering with forged identities to try to get maintainers to merge the project.
They released an in depth technical report alongside their announcement. The one thing I don’t necessarily understand here is that their eval setup permitted Internet access. OAI and Anthropic, for example, explicitly restrict Internet access, because they issue impossible tasks where they want to evaluate how the model reasons through solving the task without trying to cheat.
Meta says its AI model breached a third-party company during testing by CBS News
It was very difficult for me to find a source blog for this story, but I learned that it’s because Meta never released one. Meta spokespeople reached out to several news networks and disclosed that one of their models broke out of their training environment during training and breached an AI safety company called Irregular. Funny enough, Irregular has no blog or post-mortem on the breach, so I hope we can see one soon and see how the discovered vulnerability was exploited.
Other news
Zoomsday by Idan Levcovich
Levcovich released a blog on several vulnerabilities found by the A research team in Zoom. The research details how Zoom’s annotation protocol allowed an attacker to send a crafted annotation message that crossed a role boundary, reached a victim’s deserializer, and triggered memory-corruption bugs in the shared cross-platform annotation library. The deserialization part is interesting: according to Levcovich, Zoom has an undocumented serialization protocol with annotations to help it transcribe drawings on screen. They abused this serialization to achieve remote code execution.
I like that they included the prompts they used to help target Zoom. They decompiled the APK app for Android, and ran a threat modeling prompt, and discovered a shared library that contained the serialization protocol. Zoom issued 3 CVEs, 2 specifically for A Security and one for a separate vulnerability they found but was already disclosed to Zoom security.
TIL what Microsoft Power Pages are:
Quickly create enterprise-grade AI-driven business portals with built-in agents.
According to Fortra researchers, ExfilSquad, a new data ransomware group, exploited default configurations in Power Pages to gain malicious access and exfiltrate data from 15 victim organizations. ExfilSquad released data from 13 organizations, totaling roughly 382 GB and 27 million records. The leaked JSONL and CRM/ERP fields are consistent with Dynamics 365 exports, and Fortra found no evidence of ransomware deployment.
In the rush of pushing out as many AI products as possible, Microsoft likely didn’t consider threat models surrounding products like Power Pages. Default configurations are tough: we learned how to harden things quickly when we ran our own servers internally after years of pain. The hard part for me here is that when I procure a SaaS product or deploy a product from a cloud provider like Microsoft, I would expect more attention to detail that we trust them with securely deploying it.
🔗 Open Source
GitHub link to Vega Threat Research’s detection skills blog. Contains the full specifications and example skills to inject into your own agentic workflows.
Anton Ovrutsky’s repository to his ATEN project, featured in the State of the Art section above. The markdown and general documentation is amazing, especially the attribution section which ties an agent prompt turn to a kernel level action.
CipherRun is a Rust-based TLS scanner that combines protocol and cipher-suite analysis, vulnerability checks, compliance controls, and large-scale scanning. It tests every version of TLS from SSLv2 all the way to TLS 1.3. Some other useful features and enrichment include checking for nearly 20 vulnerabilities, JA3/JA4 hash analysis and even monitoring certificate transparency logs for alerts.
AsaEdgerunner/linux-terminal-mastery
Open-source course focused on the Linux terminal that progresses from beginner command-line fundamentals to power-user workflows. Not security-specific, but a clean resource to hand to analysts or researchers who need to get more comfortable using Linux.