A new Microsoft Security blog for your consideration.
Overview:
Last week, on Monday June 14th, 2021, a new
version of the Windows Security Events data connector reached public preview. This is
the first data connector created leveraging the new
generally available Azure Monitor Agent (AMA) and Data Collection Rules (DCR) features from the Azure
Monitor ecosystem. As any other new feature in Azure Sentinel, I
wanted to expedite the testing process and empower others in the InfoSec
community through a lab environment to learn more about it.
In this post, I will talk about the
new features of the new data connector and how to automate the
deployment of an Azure Sentinel instance with the
connector enabled, the creation and association of DCRs
and installation of the AMA on a Windows workstation. This is an
extension of a blog post I wrote, last year (2020), where I covered
the collection of Windows security events via the Log Analytics Agent (Legacy).
Recommended
Reading
I highly recommend reading the following blog posts
to learn more about the announcement of the new Azure Monitor
features and the Windows Security Events data connector:
- Azure Monitor Agent and Data
Collection Rules now generally available | Azure updates | Microsoft Azure - A powerful agent for Azure
Monitor and a simpler world of data collection; now generally available! –
Microsoft Tech Community - Connect Windows security event
data to Azure Sentinel (tabbed version) | Microsoft Docs
Azure Sentinel To-Go!?
Azure Sentinel2Go is an open-source project maintained
and developed by the Open Threat Research community to automate the deployment of an Azure
Sentinel research lab and a data ingestion pipeline to consume
pre-recorded datasets. Every environment I release through this
initiative is an environment I use and test while performing research as part
of my role in the MSTIC R&D team. Therefore, I am constantly trying to
improve the deployment templates as I cover more scenarios. Feedback
is greatly appreciated.
A
New Version of the Windows Security Events Connector?
According to Microsoft docs, the Windows Security Events connector lets you stream
security events from any Windows server (physical or virtual, on-premises or in
any cloud) connected to your Azure Sentinel workspace. After last week,
there are now two versions of this connector:
- Security
events (legacy version):
Based on the Log Analytics Agent (Usually known as the Microsoft
Monitoring Agent (MMA) or Operations Management Suite
(OMS) agent). - Windows
Security Events (new version):
Based on the new Azure Monitor Agent (AMA).
In your Azure Sentinel data connector’s view, you
can now see both connectors:
A
New Version? What is New?
Data
Connector Deployment
Besides using the Log Analytics Agent to
collect and ship events, the old connector uses the Data Sources resource from the Log Analytics Workspace resource to set the collection tier
of Windows security events.
The new connector, on the other hand, uses a combination of Data Connection Rules (DCR) and Data Connector
Rules Association (DCRA).
DCRs define what data to collect and where it should be
sent. Here is where we can set it to send data to the log analytics
workspace backing up our Azure Sentinel instance.
In order to apply a DCR to a virtual machine, one
needs to create an association between the machine and the rule. A
virtual machine may have an association with multiple DCRs, and a DCR
may have multiple virtual machines associated with it.
For more detailed information about
setting up the Windows Security Events connector with both Log Analytics Agent
and Azure Monitor Agents manually,
take a look at this document.
Data
Collection Filtering Capabilities
The old connector is not flexible enough to
choose what specific events to collect. For example, these are the
only options to collect data from Windows machines with the old connector:
All events –
All Windows security and AppLocker events.
- Common –
A standard set of events for auditing purposes. The Common event set
may contain some types of events that aren’t so common. This is because
the main point of the Common set is to reduce the volume of events to a
more manageable level, while still maintaining full audit trail capability.
- Minimal –
A small set of events that might indicate potential threats. This set does
not contain a full audit trail. It covers only events that might indicate
a successful breach, and other important events that have very low rates
of occurrence. - None –
No security or AppLocker events. (This setting is used to disable the
connector.)
According to Microsoft docs, these are
the pre-defined security event collection groups
depending on the tier set:
On the other hand, the new connector allows custom data collection via XPath queries. These
XPath queries
are defined during the creation of the data collection rule and
are written in the form of LogName!XPathQuery. Here
are a few examples:
- Collect only Security events with Event ID = 4624
Security!*[System[(EventID=4624)]]
- Collect only Security events with Event ID = 4624 or
Security Events with Event ID = 4688
Security!*[System[(EventID=4624 or EventID=4688)]]
- Collect only Security events with Event ID = 4688 and
with a process name of consent.exe.
Security!*[System[(EventID=4688)]] and *[EventData[Data[@Name=’ProcessName’]
=’C:WindowsSystem32consent.exe’]]
You can select the custom
option to select which events to stream:
Important!
Based on the new connector docs, make sure to query only Windows Security and AppLocker
logs. Events from other Windows logs, or from security logs from other
environments, may not adhere to the Windows Security Events schema and won’t be
parsed properly, in which case they won’t be ingested to your workspace.
Also, the Azure Monitor agent supports XPath
queries for XPath version 1.0 only. I recommend reading the Xpath 1.0 Limitation documentation before writing XPath Queries.
XPath?
XPath stands for XML (Extensible
Markup Language) Path language, and it is used to
explore and model XML documents as a tree of nodes. Nodes can be
represented as elements,
attributes, and
text.
In the image below, we can see a few node examples
in the XML representation of a Windows security event:
XPath Queries?
XPath queries are used to search for patterns in XML
documents and leverage path expressions and predicates
to find a node or filter specific nodes that contain a specific
value. Wildcards such as ‘*’
and ‘@’
are used to select nodes and predicates are always embedded in square brackets “[]”.
Matching any element node with ‘*’
Using our previous Windows Security event
XML example, we can process Windows Security events using the
wildcard ‘*’ at
the `Element` node level.
The example below walks
through two ‘Element’ nodes
to get to the ‘Text’
node of value ‘4688’.
You can test this basic ‘XPath’ query via
PowerShell.
- Open a PowerShell console as ‘Administrator’.
- Use the Get-WinEvent command to pass the XPath query.
- Use the ‘Logname’ parameter to define what event
channel to run the query against. - Use the ‘FilterXPath’ parameter to set the XPath query.
Get-WinEvent -LogName Security -FilterXPath ‘*[System[EventID=4688]]
Matching any attribute node with ‘@’
As shown before, ‘Element’ nodes can contain ‘Attributes’ and we can
use the wildcard ‘@’
to search for ‘Text’
nodes at the ‘Attribute’
node level. The example below extends the
previous one and adds a filter to search for a specific ‘Attribute’ node that
contains the following text: ‘C:WindowsSystem32cmd.exe’.
Once again, you can test the XPath query via
PowerShell as Administrator.
$XPathQuery = “*[System[EventID=4688]] and *[EventData[Data[@Name=’ParentProcessName’]=’C:WindowsSystem32cmd.exe’]]”
Get-WinEvent -LogName Security -FilterXPath $XPathQuery
Can I Use XPath Queries in Event Viewer?
Every time you add a filter through the Event
Viewer UI, you can also get to the XPath query representation of the
filter. The XPath query is part of a QueryList node which
allows you to define and run multiple queries at once.
We can take our previous example where we searched
for a specific attribute and run it through the Event Viewer Filter
XML UI.
<QueryList>
<Query Id="0" Path="Security">
<Select Path="Security">*[System[(EventID=4688)]] and *[EventData[Data
[@Name='ParentProcessName']='C:WindowsSystem32cmd.exe']]</Select>
</Query>
</QueryList>
Now that we have covered some of the main changes
and features of the new version of the Windows Security Events data connector,
it is time to show you how to create a lab environment for you to test your own
XPath queries for research purposes and before pushing
them to production.
Deploy Lab Environment
- Identify the right Azure resources
to deploy. - Create deployment template.
- Run deployment template.
Identify the Right Azure Resources to Deploy
As mentioned earlier in this post, the old connector uses the Data Sources resource from the Log Analytics Workspace resource to set the collection tier of Windows
security events.
This is the Azure Resource Manager (ARM) template I
use in Azure-Sentinel2Go to set it up:
Azure-Sentinel2Go/securityEvents.json
at master · OTRF/Azure-Sentinel2Go (github.com)
Data
Sources Azure Resource
{
"type": "Microsoft.OperationalInsights/workspaces/dataSources",
"apiVersion": "2020-03-01-preview",
"location": "eastus",
"name": "WORKSPACE/SecurityInsightsSecurityEventCollectionConfiguration",
"kind": "SecurityInsightsSecurityEventCollectionConfiguration",
"properties": {
"tier": "All",
"tierSetMethod": "Custom"
}
}
However, the new connector uses a combination of Data Connection Rules (DCR) and Data Connector Rules Association (DCRA).
This is the ARM template I use to
create data collection rules:
Azure-Sentinel2Go/creation-azureresource.json at master ·
OTRF/Azure-Sentinel2Go (github.com)
Data
Collection Rules Azure Resource
{
"type": "microsoft.insights/dataCollectionRules",
"apiVersion": "2019-11-01-preview",
"name": "WindowsDCR",
"location": "eastus",
"tags": {
"createdBy": "Sentinel"
},
"properties": {
"dataSources": {
"windowsEventLogs": [
{
"name": "eventLogsDataSource",
"scheduledTransferPeriod": "PT5M",
"streams": [
"Microsoft-SecurityEvent"
],
"xPathQueries": [
"Security!*[System[(EventID=4624)]]"
]
}
]
},
"destinations": {
"logAnalytics": [
{
"name": "SecurityEvent",
"workspaceId": "AZURE-SENTINEL-WORKSPACEID",
"workspaceResourceId": "AZURE-SENTINEL-WORKSPACERESOURCEID"
}
]
},
"dataFlows": [
{
"streams": [
"Microsoft-SecurityEvent"
],
"destinations": [
"SecurityEvent"
]
}
]
}
}
One additional step in the setup of the new
connector is the association of the DCR with Virtual Machines.
This is the ARM template I use
to create DCRAs:
Azure-Sentinel2Go/association.json at master ·
OTRF/Azure-Sentinel2Go (github.com)
Data
Collection Rule Associations Azure Resource
{
"name": "WORKSTATION5/microsoft.insights/WindowsDCR",
"type": "Microsoft.Compute/virtualMachines/providers/dataCollectionRuleAssociations",
"apiVersion": "2019-11-01-preview",
"location": "eastus",
"properties": {
"description": "Association of data collection rule. Deleting this association will break the data collection for this virtual machine.",
"dataCollectionRuleId": "DATACOLLECTIONRULEID"
}
}
What
about the XPath Queries?
As shown in the previous section, the XPath query
is part of the “dataSources”
section of the data collection rule resource. It is defined under the ‘windowsEventLogs’ data
source type.
"dataSources": {
"windowsEventLogs": [
{
"name": "eventLogsDataSource",
"scheduledTransferPeriod": "PT5M",
"streams": [
"Microsoft-SecurityEvent"
],
"xPathQueries": [
"Security!*[System[(EventID=4624)]]"
]
}
]
}
Create Deployment Template
We can easily add all those
ARM templates to an ‘Azure Sentinel & Win10 Workstation’ basic template. We just need to make
sure we install the Azure
Monitor Agent instead of the Log
Analytics one, and enable the system-assigned managed identity in the Azure VM.
Template
Resource List to Deploy:
- Azure Sentinel Instance
- Windows Virtual Machine
- Azure Monitor Agent Installed.
- System-assigned managed identity Enabled.
- Data Collection Rule
- Log Analytics Workspace ID
- Log Analytics Workspace Resource ID
- Data Collection Rule Association
- Data Collection Rule ID
- Windows Virtual Machine Resource Name
The following ARM template can be used for our
first basic scenario:
Azure-Sentinel2Go/Win10-DCR-AzureResource.json at master ·
OTRF/Azure-Sentinel2Go (github.com)
Run Deployment Template
You can deploy the ARM template via a “Deploy to Azure” button or via Azure CLI.
“Deploy to Azure” Button
- Browse
to Azure Sentinel2Go repository - Go
to grocery-list/Win10/demos. - Click
on the “Deploy to
Azure” button next to “Azure Sentinel + Win10 + DCR (DCR Resource)”
- Fill
out the required parameters:
- adminUsername:
admin user to create in the Windows workstation. - adminPassword:
password for admin user. - allowedIPAddresses:
Public IP address to restrict access to the lab environment. - Wait
5-10 mins and your environment should be ready.
Azure CLI
- Download demo template.
- Open a terminal where
you can run Azure CLI from (i.e. PowerShell). - Log in to your Azure
Tenant locally.
az login
- Create Resource Group
(Optional)
az group create -n AzSentinelDemo -l eastus
- Deploy ARM template
locally.
az deployment group create –f ./ Win10-DCR-AzureResource.json -g MYRESOURCRGROUP –adminUsername MYUSER –adminPassword MYUSERPASSWORD –allowedIPAddresses x.x.x.x
- Wait 5-10 mins and your
environment should be ready.
Whether you use the UI or the CLI, you can monitor
your deployment by going to Resource Group > Deployments:
Verify
Lab Resources
Once your environment is deployed successfully, I
recommend verifying every resource that was deployed.
Azure
Sentinel New Data Connector
You will see the Windows Security Events (Preview) data
connector enabled with a custom Data
Collection Rules (DCR):
If you edit the custom DCR, you will see the XPath
query and the resource that it got associated with. The image below shows the
association of the DCR with a machine named workstation5.
You can also see that the data collection is set to custom and, for this
example, we only set the event stream to collect events with Event ID 4624.
Windows Workstation
I recommend to RDP to the Windows Workstation by
using its Public IP Address. Go to your resource group and select the Azure VM.
You should see the public IP address to the right of the screen.
This would generate authentication events which will be captured by
the custom DCR associated with the endpoint.
Check Azure Sentinel Logs
Go back to your Azure Sentinel, and you should
start seeing some events on the Overview page:
Go to Logs
and run the following KQL query:
SecurityEvent
| summarize count() by EventID
As you can see in the image below, only events with
Event ID 4624 were collected by the Azure Monitor Agent.
You might be asking yourself, “Who would only want
to collect events with Event ID 4624 from a Windows endpoint?”.
Believe it or not, there are network environments where due to bandwidth
constraints, they can only collect certain events. Therefore, this custom
filtering capability is amazing and very useful to cover more use cases and
even save storage!
Any Good XPath Queries Repositories in the InfoSec
Community?
Now that we know the internals of the new connector
and how to deploy a simple lab environment, we can test multiple XPath queries
depending on your organization and research use cases and bandwidth
constraints. There are a few projects that you can use.
Palantir WEF Subscriptions
One of many repositories out there that
contain XPath queries is the ‘windows-event-forwarding’ project from Palantir. The XPath queries are Inside of the Windows
Event Forwarding (WEF) subscriptions. We could take all the subscriptions
and parse them programmatically to extract all the XPath
queries saving them in a format that can be used to be part of the
automatic deployment.
You can run the following steps in this document
available in Azure Sentinel To-go and extract XPath queries from the Palantir
project.
Azure-Sentinel2Go/README.md at master · OTRF/Azure-Sentinel2Go
(github.com)
OSSEM Detection Model + ATT&CK Data Sources
From a community perspective, another great
resource you can use to extract XPath Queries from is the Open
Source Security Event Metadata (OSSEM) Detection Model (DM) project. A community driven effort to help
researchers model attack behaviors from a data perspective and
share relationships identified in security events across several operating
systems.
One of the use cases from this initiative
is to map all security events in the project to the new ‘Data Sources’ objects provided by the MITRE ATT&CK
framework. In the image below, we can
see how the OSSEM DM project provides an interactive document (.CSV) for researchers to explore
the mappings (Research output):
One of the advantages of this project over others is
that all its data relationships are in YAML format which makes
it easy to translate to others formats. For example, XML. We can use
the Event IDs defined in each data
relationship documented in OSSEM DM and create XML files with XPath queries
in them.
Exploring
OSSEM DM Relationships (YAML Files)
Let’s say we want to use relationships related to scheduled jobs in Windows.
Translate
YAML files to XML Query Lists
We can process all the YAML files and export the
data in an XML files. One thing that I like about this
OSSEM DM use case is that we can group the XML files by ATT&CK data sources. This can help organizations
organize their data collection in a way that can be mapped to detections or
other ATT&CK based frameworks internally.
We can use the QueryList format to document all ‘scheduled jobs relationships‘
XPath queries in one XML file.
I like to document my XPath
queries first in this format because it expedites the validation
process of the XPath queries locally on a Windows endpoint. You can use that
XML file in a PowerShell command to query Windows Security events and make
sure there are not syntax issues:
[xml]$scheduledjobs = get-content .scheduled-job.xml
Get-WinEvent -FilterXml $scheduledjobs
Translate
XML Query Lists to DCR Data Source:
Finally, once the XPath queries have been
validated, we could simply extract them from the XML files and put them in a
format that could be used in ARM templates to create DCRs. Do you
remember the dataSources property
of the DCR Azure resource we talked about earlier? What if we could get
the values of the windowsEventLogs data
source directly from a file instead of hardcoding them in an ARM template? The
example below is how it was previously being hardcoded.
"dataSources": {
"windowsEventLogs": [
{
"name": "eventLogsDataSource",
"scheduledTransferPeriod": "PT5M",
"streams": [
"Microsoft-SecurityEvent"
],
"xPathQueries": [
"Security!*[System[(EventID=4624)]]"
]
}
]
}
We could use the XML files created after processing
OSSEM DM relationships mapped to ATT&CK data sources and creating
the following document. We can pass the URL of the document as a parameter in
an ARM template to deploy our lab environment:
Azure-Sentinel2Go/ossem-attack.json at master ·
OTRF/Azure-Sentinel2Go (github.com)
Wait!
How Do You Create the Document?
The OSSEM team is contributing and
maintaining the JSON file from the previous section in
the Azure Sentinel2Go repository. However, if you want to
go through the whole process on your own, Jose Rodriguez (@Cyb3rpandah) was kind enough to write every single
step to get to that output file in the following blog post:
Ok, But, How Do I Pass the JSON
file to our Initial ARM template?
In our initial ARM template, we had the XPath query as an
ARM template variable as shown in the image below.
We could also have it as a template parameter.
However, it is not
flexible enough to define multiple DCRs or even update the whole DCR Data
Source object (Think about future coverage beyond Windows
logs).
Data Collection Rules – CREATE API
For more complex use cases, I would use the DCR Create API. This can be executed via a PowerShell script
which can also be used inside of an ARM template via deployment scripts. Keep in mind that, the deployment script resource requires
an identity to execute the script. This managed identity of type user-assigned can be created at
deployment time and used to create the DCRs programmatically.
PowerShell Script
If you have an Azure Sentinel instance without the
data connector enabled, you can use the following PowerShell script to
create DCRs in it. This is good for testing and it also works in ARM templates.
Keep in mind, that you would need to have a
file where you can define the structure of the windowsEventLogs data source object
used in the creation of DCRs. We created that in the previous section remember?
Here is where we can use the OSSEM Detection
Model XPath Queries File 😉
Azure-Sentinel2Go/ossem-attack.json at master ·
OTRF/Azure-Sentinel2Go (github.com)
FileExample.json
{
"windowsEventLogs": [
{
"Name": "eventLogsDataSource",
"scheduledTransferPeriod": "PT1M",
"streams": [
"Microsoft-SecurityEvent"
],
"xPathQueries": [
"Security!*[System[(EventID=5141)]]",
"Security!*[System[(EventID=5137)]]",
"Security!*[System[(EventID=5136 or EventID=5139)]]",
"Security!*[System[(EventID=4688)]]",
"Security!*[System[(EventID=4660)]]",
"Security!*[System[(EventID=4656 or EventID=4661)]]",
"Security!*[System[(EventID=4670)]]"
]
}
]
}
Run
Script
Once you have a JSON file similar to the one in the previous section, you
can run the script from a PowerShell console:
.Create-DataCollectionRules.ps1 -WorkspaceId xxxx -WorkspaceResourceId xxxx -ResourceGroup MYGROUP -Kind Windows -DataCollectionRuleName WinDCR -DataSourcesFile FileExample.json -Location eastus –verbose
One thing to remember is that you can only have 10
Data Collection rules. That is
different than XPath queries inside of one DCR. If you attempt to create more than 10 DCRs, you
will get the following error message:
ERROR
VERBOSE: @{Headers=System.Object[]; Version=1.1; StatusCode=400; Method=PUT;
Content={"error":{"code":"InvalidPayload","message":"Data collection rule is invalid","details":[{"code":"InvalidProperty","message":"'Data Sources. Windows Event Logs' item count should be 10 or less. Specified list has 11 items.","target":"Properties.DataSources.WindowsEventLogs"}]}}}
Also, if you have duplicate XPath queries in one DCR,
you would get the following message:
ERROR
VERBOSE: @{Headers=System.Object[]; Version=1.1; StatusCode=400; Method=PUT;
Content={"error":{"code":"InvalidPayload","message":"Data collection rule is invalid","details":[{"code":"InvalidDataSource","message":"'X Path Queries' items must be unique (case-insensitively).
Duplicate names:
Security!*[System[(EventID=4688)]],Security!*[System[(EventID=4656)]].","target":"Properties.DataSources.WindowsEventLogs[0].XPathQueries"}]}}}
ARM Template: DeploymentScript Resource
Now that you know how to use a PowerShell script to
create DCRs directly to your Azure Sentinel instance, we can use it inside of
an ARM template and make it point to the JSON file that contains all the XPath
queries in the right format contributed by the OSSEM DM project.
This is the template I use to put it all together:
Azure-Sentinel2Go/Win10-DCR-DeploymentScript.json at master ·
OTRF/Azure-Sentinel2Go (github.com)
What about the DCR Associations?
You still need to associate the DCR with a virtual
machine. However, we can keep doing that within the template leveraging the DCRAs Azure resource linked template inside of the main
template. Just in case you were wondering how I call the linked template from
the main template, I do it this way:
Azure-Sentinel2Go/Win10-DCR-DeploymentScript.json at master ·
OTRF/Azure-Sentinel2Go (github.com)
How Do I Deploy the
New Template?
The same way how we deployed the initial one. If you want the Easy Button , then simply
browse to the URL below and click on the blue button highlighted in the image
below:
Link: Azure-Sentinel2Go/grocery-list/Win10/demos at master ·
OTRF/Azure-Sentinel2Go (github.com)
Wait 5-10 mins!
Enjoy it!
That’s it! You now know two ways to deploy and
test the new data connector and Data
Collection Rules features with XPath queries capabilities. I hope this
was useful. Those were all my notes while testing and developing templates
to create a lab environment so that you could also expedite the testing process!
Feedback is greatly appreciated! Thank you to the OSSEM team and the Open Threat Research
(OTR) community for helping us operationalize the research they share with
the community! Thank you, Jose Rodriguez.
Demo Links
- Azure-Sentinel2Go/grocery-list/Win10/demos at master ·
OTRF/Azure-Sentinel2Go (github.com) - Azure-Sentinel2Go/azure-sentinel/linkedtemplates/data-collection-rules
at master · OTRF/Azure-Sentinel2Go (github.com) - Azure-Sentinel2Go/azure-sentinel/linkedtemplates/data-collection-rules/rules
at master · OTRF/Azure-Sentinel2Go (github.com) - Azure-Sentinel2Go/ossem-attack.json at master ·
OTRF/Azure-Sentinel2Go (github.com)
References
- XPath Tutorial (w3schools.com)
- Configure data collection for the
Azure Monitor agent (preview) – Azure Monitor | Microsoft Docs - Data Collection Rules in Azure
Monitor (preview) – Azure Monitor | Microsoft Docs - Connect Windows security event
data to Azure Sentinel (tabbed version) | Microsoft Docs
- Connect Windows security event
data to Azure Sentinel (tabbed version) | Microsoft Docs - Microsoft.Insights/dataCollectionRuleAssociations
– ARM template reference | Microsoft Docs - Advanced XML filtering in the
Windows Event Viewer – Microsoft Tech Community - Consuming Events (Windows Event
Log) – Win32 apps | Microsoft Docs
- Azure Monitor agent overview –
Azure Monitor | Microsoft Docs - A powerful agent for Azure
Monitor and a simpler world of data collection; now generally available! –
Microsoft Tech Community - Data Collection Rules – REST API
(Azure Monitor) | Microsoft Docs - Data Collection Rule Associations
– REST API (Azure Monitor) | Microsoft Docs - XML Path Language (XPath) (w3.org)
- Managed identities for Azure
resources | Microsoft Docs - Configure managed identities
using the Azure portal – Azure AD | Microsoft Docs