Detection Field Manual #2 - What are Detection Rules?
The fundamental concept behind all blue team operations
The fundamental building block for detecting malicious activity inside an environment is detection rules. Detection Engineering functions, such as a Security Operations Center (SOC) or a Detection and Response team, maintain a ruleset to detect malicious activity within their environment.
As we dig deeper into the technical aspects of Detection Engineering, we first need to explore how humans scale their threat expertise into logging systems.
Detection Engineering Interview Questions:
What are detection rules?
Why do detection rules matter for a security team?
How does SIEM indexing work, and can you provide examples of how a SIEM processes and searches logs?
There is no rule without telemetry
A detection rule at its fundamental level is a search query against a logging system that contains logs. In the security industry, most threat detection teams operate a Security Information and Event Management (SIEM) logging system. Detection teams gather telemetry from Assets in the form of logs and send them to the SIEM.
Logs are typically generated when an Asset records an Event. In the Operating System world, such as Windows or Linux, an event can be:
A new user is being added to the system.
A process or a user created, modified, or deleted a file.
A user runs or executes a file that creates a new Process.
Recording everyday activity can be cumbersome, so for threat detection, some events are more desirable than others. For example, Sysinternals’ Sysmon records 27 different types of security events that can happen on Windows. Event Tracing for Windows (ETW) providers can range in the 10s of thousands, and you need to subscribe to providers who can provide relevant security events to you.
When a log enters a SIEM, it is stored on disk and typically sent to an “index.” An index is a structured search layer built on top of stored logs, allowing for fast lookups by mapping field values (such as domains, IPs, or processes) to logs that contain them.
For example, in Splunk, if you specify your Assets to send a log to the “WebApp” index, it’ll store those raw logs alongs with some metadata. During indexing, the SIEM parses and enriches each log with structured fields. If that parsing fails, or there are errors on ingestion, some fields may be missing.
When you search a SIEM, you generally provide an index and a query. The query can contain basic strings, such as google.com , and the SIEM should send relevant logs that contain google.com. But we aren’t here to do simple things; we are here to find bad guys, so how do you create rules to do so?
Rules encapsulate Human knowledge
Remember, the primary function of a rule is to store a search query. This search query, when issued against the SIEM, MAY return results. If results are returned, it could be indicative of malicious activity.
The above example shows a simplified workflow on how this works. A Detection Engineer asks the SIEM to send them any logs that contain maliciousdomain.com on index:dns. The SIEM utilizes the index to identify relevant logs, thereby avoiding full storage scans. Once the SIEM and underlying indexing technology find matches, it retrieves them from storage along with the associated metadata.
But what’s the problem with this approach? It’s the human! Here’s why:
Humans can’t possibly remember every search query needed to find malicious activity
Even if the above bullet were true, if the list is N long, are you going to search N queries for results sequentially?
When your query returns a malicious result, do you stop querying to investigate? What if more malicious activity is occurring?
To scale this problem, we store queries associated with threat actor activity as Rules.
In the simplified example above, a ruleset is a set of rules built by a Detection Engineer. A querying service retrieves the set of rules and executes the queries within it.
Detection rules can execute on a fixed schedule (e.g., every 5 or 30 minutes) or in near real-time as logs are ingested. This varies by the SIEM technology the organization uses.
The same steps happen in the SIEM; if the query contains valid search parameters, such as an index that exists, it’ll look for it in storage. Once retrieved, results are sent to the analyst. This can be an alert if results are available, or it can be nothing if no results are found.
Conclusion
Understanding why rules matter as a scaling mechanism for a security team is an essential concept in threat detection. Encapsulating human knowledge inside dozens to hundreds of detection rules helps relieve analysts of the operational burden of searching for malicious activity. Additionally, it frees up analysts so they can orient themselves, tune their knowledge, and share it with others.
A SIEM is a critical component to many blue team functions, and seeing how logs are stored in a technology that can ingest logs, perform fast searches, and present other necessary metadata makes finding threat actor activity nearly instantaneous.
No system is perfect, though, and in the upcoming issues we’ll be studying types of rules (with some real-world examples) and comparing and contrasting alerting strategies.
Stay tuned!
Icon attribution
https://www.flaticon.com/free-icon/database-storage_2906206?term=database&related_id=2906206