DEW #165 - Starved & Neglected Rules, Agentic IR Notebooks and Hacker Summer Camp
my body is not ready but I did book a cabana :3
Welcome to Issue #165 of Detection Engineering Weekly!
✍️ Musings from the life of Zack:
I will be at Hacker Summer Camp next week! I am hosting three events: two panels & a happy hour. I put the details at the end of the post in lieu of the Open Source Section. Come find me and grab a sticker!
I spent time over the weekend at a lakehouse, and besides the pretty water views, there’s an eerie but beautiful sound I listen for at night. The Loon Wail is such a unique sound, and it’s something my kids get excited about every time they hear it.
I’ve switched over nearly all of my coding agents to Pi. The customization and the ability to switch between model providers gives me Vim vibes.
detections.ai Enterprise is live. Private, coverage and context aware
A new threat drops and you need to move quickly. Run coverage analysis across your whole detection stack, build detections tuned directly for your environment, and deploy them back. In minutes, not days.
But coverage isn’t a one-time win. Detections drift, IOCs go stale, duplicate rules pile up. Our AI agents catch it while you sleep, so nothing slips through unnoticed, and your team spends its time on real threats, not upkeep.
💎 Detection Engineering Gem 💎
Detections that run but can’t see by Nikhil Satyakrishna
SIEM and detection readiness is a topic I’m hearing more and more about in the research, security product, and internal operations spaces. I seem to link back to my Field Manual post a lot, but the line “There is No Rule without Telemetry” still holds true, but my post fails to address the underlying resilience gotchas. In this post, Satyakrishna addresses these issues and releases an open-source tool to demonstrate how to check for resilience failures.
The four failure modes worth keeping are:
Disconnected: index pattern matches nothing
Starved: sources exist but are stale or empty
Missing fields: the field the rule filters on stopped being populated
Ingest-lag blind window: events land after the rule’s window has already passed
If you asked me two years ago where the most important failure mode to check for is, it would be “starved”. A log source that doesn’t ship telemetry should be a red flag, but it’s much harder to check for than one would think. Is it not shipping telemetry because there is no telemetry to ship, or is there an underlying issue with the ingest pipeline?
Missing fields are a hard one to crack as well. You’ll need to understand the shape of the logs you are sending in, and make sure that if a log contains a missing field, it’s not due to a broken log source, an upstream provider changing the log envelope, or it’s expected behavior. As I’ve said before, we are becoming closer to SREs than we think :).
I’ve covered xcal’s work for the ingest-lag blind window issue using watermarks, but it’s nice to see another take on this on a log-agnostic use case:
Fraction of events a rule catches as source ingest lag grows, for a rule with a 6-minute lookback running every 5 minutes. The line is the formula, the open points are the simulation, and the shaded area is coverage lost with no error and no alert.
Satyakrishna released an Elastic-specific tool to check for the scenarios listed above plus two more, dubbed deadair. They also linked Elastic’s SIEM Readiness page, so it’s clear to me that research in the detection space is focusing more and more on detection readiness and resilience.
🔬 State of the Art
Agentic incident response notebooks by Kyrre Wahl Kongsgård
I’ve worked on hundreds of incidents at this point in my career, and it almost always falls into the same pattern. A Google Sheet gets shared, and by hour two everyone has 20+ tabs open: SIEM in one, threat intel platform in another, an asset inventory tool in a third, and the infamous scratchpad notes scattered in another. It makes sense, right? Security tools don’t do a great job of collating data, and giving yourself and your team more control over content curation means you need tools everyone can quickly orient to, to visualize the current state of the incident.
Kongsgård & the DNB’s Cyber Defense Center (authors from a previous Gem and building on work I featured almost 60 issues ago) built a super interesting agentic IR workflow on Marimo, where the notebook itself is the case file. Marimo beats Jupyter for this job for reasons that matter specifically to IR, such as Jupyter cells can run out of order, and an output can depend on state from a cell you already deleted, which isn’t helpful when you are trying to maintain chain of evidence or investigative steps.
Marimo gives responders a much more reactive experience. You change the host filter or the time window and every dependent query, join, and chart recomputes from the code actually on screen, versus pressing the play button on every subsequent Jupyter cell. Because the file is plain Python, the same notebook works as the analyst’s live workspace, so it’s great for source control.
They have several embedded examples that you can follow along, but my favorite part is their agentic loop for managing investigations via skills rather than building notebooks for each alert type:
The agent loop combines skills to manage the frontend investigation panel, specific domain skills on triaging and hunting through telemetry, and creating or maintaining sessions inside the investigative notebook. The domain skills help remove the complexity around dozens of notebooks per investigation because you can abstract the investigative patterns into markdown files versus code itself.
The Arguments Against Open Source AI are Very Bad by Tom Bedor
Last week’s issue contained stories around Kimi K3 and the Hugging Face/OpenAI model “breach”. It was brilliant timing for that issue: a Chinese company released a Frontier Labs-comparable open-weight model, and Hugging Face disclosed that they needed an open-weight model during their IR investigation since Frontier Labs APIs wouldn’t accept their investigative prompts due to cyber guardrails.
Bedor’s blog here is a reaction to the Kimi K3 blog and subsequent public reaction, where several prominent executives and leaders argue that open-weight models are too dangerous to release to the world. OpenAI’s Dean Ball specifically said:
One probable outcome of an open-weight-model-dominant world is full AI communism... rather than a market product, AI is a “public good”
Without open-weight models, could Hugging Face have reacted fast or efficiently enough? History is also not helping the argument for restricting open-weight access. Bedor cites several stories about the government trying to limit access to open-source software, such as SSL or PGP, and ultimately failing.
IMHO, his strongest point is that limiting who can inspect and run a model only serves attackers, since responsible actors are the ones who need unrestricted access to patch and investigate. HF’s own postmortem says the attacker was bound by no usage policy, while the defenders were the ones blocked by guardrails.
My advice for defenders: you should strongly consider open-weight models for investigation, especially if you don’t have access to lower guardrail models. It’s a discussion to have within your team all the way up to your CSO.
Accelerating EDR Evasion with LLM-Driven Analysis by Adam Chester
The Ralph Wiggum Loop is an agentic looping strategy where you shove a prompt into a simple infinite loop and have your coding agent work through the prompt indefinitely. It looks something like this:
The idea is that you assume your coding agent's prompts reach an eventual end, whether it’s through failure or through it completing its goal. It’s a harness that works so well that you see coding agents implement variations of the technique. This is especially useful when you can afford and want to run loops and trust that your agents can handle the goal prompt you have in mind, especially if it’s long-running analysis work.
This is where Chester’s blog falls nicely into place when you have goals surrounding cybersecurity analysis. In this post, Chester implements the Ralph Wiggum Loop to reverse-engineer Palo Alto’s Cortex EDR to find evasion opportunities. From a red teamer perspective, this is super helpful for several reasons:
You farm out compute and mental load to a coding agent to help give you leads on evasion techniques
The coding agent has access to the same tooling as you do, and can provide proof of its findings for you to verify
The coding agent has context on previous runs, so that it doesn’t go down rabbit holes that it already went down, so it can focus on net new attack surfaces
The specific architecture revolves around maintaining three different markdown files:
REPORT.md – A markdown file used by the running agents to surface key findings for human review.
STATE.md – A state-file which each agent can use to track key events during analysis.
CODEMAP.md – Allows each agent to store references to areas of disassembly which are interesting or critical for analysis, helping increase velocity during later agent iterations.
AGENTS.md – A set of instructions to tell the model how to use the above files.
They found several bypasses in two separate buckets: extraction and evasion of loaded detection rules & ML models and evasion of the EDR functionality itself. The agent found, decrypted, and displayed thousands of YARA rules, which can be used in a separate harness to build evasions. It also found an evasion for Cortex’s user-mode DLL, which is used to enrich on-host telemetry and provide some level of hooking for prevention.
As far as vulnerability research is ahead of the game for agentic coding, I’d love to see more on the blue team side. A Ralph Wiggum Loop for threat hunting or threat detection seems like a good use of time and tokens!
Secondary Sponsor: Rilevera
Make Your Detection Program More Resilient
Detection-as-Code is the core foundation for a resilient detection program.
Rilevera provides Detection-as-Code in the click of a button and gives you access to:
• Versioning & one-click rollback
• Peer review workflow
• Detection rule validation & efficacy reporting
• MITRE alignment mapping
Find us at Black Hat 2026 to learn more, or start a free trial now!
☣️ Threat Landscape
LG to Ban Residential Proxies from Smart TV Apps by Brian Krebs
The TV manufacturer LG recently announced that it’s banning apps that fail to remove bundled residential proxy packages inside their app store. I’ve featured several stories related to residential proxies on this newsletter, including how botnet operators infect end-user devices and add victim nodes to botnets that get rented out to these proxy networks. This ban, according to Krebs, comes right after he posted a story citing Spur’s research leading to the NetNut botnet takedown.
What I Learned from Sitting in on a Scattered Spider Sentencing by John Fitzpatrick
This blog is the first time I’ve ever seen a security researcher sit in during a court hearing and sentencing to glean technical details on attacker tradecraft. Fitzpatrick sat in the sentencing hearing for two members of Scattered Spider who were responsible for the TfL breach.
The sad and maybe boring detail is around the initial access vector. Prosecutors walked through a compromised supply-chain vendor that built niche infrastructure software for TfL, from which the attackers pulled 857 rows of user records and credentials. Those credentials dated back to 2022 and sat in the criminal ecosystem for roughly two years before being used for the breach, which resulted in a months‑long outage and tens of millions in impact.
The probe that came 17 days before the CVE by HoneyLabs
As far as I can tell, there are two ways to find zerodays: research them yourself, or wait for an attacker to throw exploits at something you control and record that traffic. This is what HoneyLabs did with their global honeypot network, and this blog is especially interesting because it dives into their architecture around finding zerodays. They specifically recorded zero-day in cPanel scanning traffic 17 days before a CVE was issued, where 16 requests to one login path from a single IP, then silence, followed by mass scanning once the advisory dropped.
Their setup on building signatures is interesting. They only count a “pre‑CVE probe” if the signature is specific to a vulnerable code path, the reference date is the earliest public signal (NVD entry or an exploit template found in a project like Nuclei), and the traffic predates both their own sensor deployment and any public exploit chatter. They also throw away anything from noisy scanners or IPs that touch more than 50 different paths. After all that filtering, just 3 out of 1,697 CVEs in their corpus still show a genuine early probe.
Detection primitives for eBPF rootkits by Lorenzo Susini and Matt Muir
~ Note: Datadog is my current place of work, and Lorenzo & Matt are my colleagues! ~
My colleagues Lorenzo and Matt published research on common techniques used by modern eBPF rootkits. For those unfamiliar with eBPF, it’s a Linux kernel technology that lets small, sandboxed programs run inside the kernel to observe or influence system behavior. Its main use cases have been observability and networking, and it also has security applications for monitoring malware.
The same technology used for security can also be abused for malware. Lorenzo and Matt explore three malware families that use different techniques to evade detection. The challenge is that once a rootkit is loaded, the same eBPF instrumentation used to find security threats may be blind to its execution. At Datadog, we detect these programs by inspecting the load-time fingerprint, such as rare helper calls, program type, and other metadata, including attach type, name, tag, and map references.
🕵️♂️ Hacker Summer Camp
I’m so excited to be back at Hacker Summer Camp after taking a few years off! I have three events that I am hosting, so if you are around during these date and times, come hang out and say hello!
🍻 Detection & Response Happy Hour - Tuesday, August 4 5pm-8pm
I am hosting a pre-BlackHat Happy Hour with friends and co-sponsors Cotool and Nebulock. Food and drink will be provided, and I’ve been diligent on not letting sales people in ;)
🪖 SIEM Under Siege - Who Will Survive? - Wednesday, August 5, 1pm-3pm
I will be at Decibel’s Game Day event hosting a panel with Jack Naglieri & Damien Lewke in an ESPN-style segment on SIEM. I have some great content already, and my hope is it won’t be a business-as-usual security discussion, and it will be way more spicy.
The Modern Detection Engineer @ Blue Team Village - Friday, August 7, 345pm-445pm
I’m excited to announce that Detection Engineering Weekly is SPONSORING Blue Team Village this year! I’ll be there handing out stickers and talking shop with conference-goers, and on Day 1 of the Village, I’m hosting a panel with some excellent guests in the detection engineering space.