Microsoft Advice for incident responders on recovery from systemic identity compromises

As Microsoft alongside our industry partners and the security community continues to investigate the extent of the Solorigate attack, our goal is to provide the latest threat intelligence including IOCs and guidance across our products and solutions to help the community fight back against, harden your infrastructure, and begin to recover from this attack of unprecedented scale. As new information becomes available, we will make updates to this article.

This blog will outline lessons learned from this and other incident response to date in on-premises and cloud environments. This latest guidance is for customers looking to re-establish trusted identities for credentials that are suspected of compromise by Solorigate malware.

This article is intended to give experienced incident responders some advice on techniques to consider when helping an organization respond to a suspected systemic identity compromise, like we’re seeing in some victims of the Solorigate malware, based on our experience in the field in similar scenarios. Re-establishing trust in the organization’s on-premises and cloud environments with minimal business impact requires in-depth investigation and an understanding of potential methods of persistence. While not meant to cover every possible scenario, this guidance is intended to summarize our experience with similar customer breaches and will be updated if we learn of new information that would help with successful recovery. Please review the resources referenced at the end of this article for additional information. This information is provided as-is and constitutes generalized guidance; the ultimate determination about how to apply this guidance to your IT environment and tenant(s) must consider your unique environment and needs, which each Customer is in the best position to determine.

The Solorigate investigation referenced in this guidance is ongoing at the time of publication and our teams continue to act as first responders to these attacks. As new information becomes available, we will make updates through our Microsoft Security Response Center (MSRC) blog.

Overview of the intrusion

As described in this Microsoft blog post, the hallmarks of this actor’s activity include, but are not limited to, the following techniques that are likely to result in systemic identity compromise:

An intrusion through malicious code in the SolarWinds Orion product. This results in the attacker gaining a foothold in the network, which the attacker can use to gain elevated credentials. Microsoft Defender now has detections for these files. Read our in-depth technical analysis of the Solorigate malware.
An intruder using administrative permissions (acquired through an on-premises compromise) to gain access to an organization’s trusted SAML token-signing certificate. This enables them to forge SAML tokens to impersonate any of the organization’s existing users and accounts, including highly privileged accounts.
Anomalous logins using the SAML tokens signed with a compromised token-signing certificate, which can be used against any on-premises resources (regardless of identity system or vendor) as well as against any cloud environment (regardless of vendor) because they have been configured to trust the certificate. An organization may miss the use of illegitimate SAML tokens because they are signed with a legitimate certificate.
The use of highly privileged accounts (acquired through the technique above or other means) to add illegitimate credentials to existing application service principals, enabling the attacker to call APIs with the permission assigned to that application.

Overview of response objectives

Organizations that have experienced systemic identity compromise need to start recovery by re-establishing trustworthy communications. This will enable effective triage and coordination of business operations recovery.

Many organizations have complex internal and external interdependencies. Core business processes and applications in an organization are likely to be temporarily impacted during recovery efforts until trust within your environment is re-established. Microsoft recommends that Incident Responders establish secure communications with key organizational personnel as the first step toward organizational recovery. If your investigation indicates that the attacker has used techniques outside of identity compromise at lower levels of your organizations’ infrastructure, such as hardware or firmware attacks, you will need to address those threats to reduce the risk of re-compromise.

Response objectives in approximate order:

Establish secure communications for personnel key to the investigation and response effort.
Investigate the environment for persistence and initial access point, while establishing continuous monitoring operations during recovery efforts.
Regain and retain administrative control of your environment and remediate or block possible persistence techniques and initial access exploits.
Improve posture by enabling security features and capabilities following best practice recommendations.

We recommend that incident responders review and digest the entirety of this guidance before taking action, as the specific order of actions taken to achieve the response objectives is very situational and depends heavily on the results (and completeness) of investigation and the business constraints of the specific organization. The following sections describe the incident Response techniques we recommend you consider for each of the above objectives.

Establish secure communications and productivity

Successful response requires being able to communicate without the attacker eavesdropping on your communications. Until you have achieved assurance in the privacy of your communications on your current infrastructure, use completely isolated identities and communication resources to coordinate your response and discuss topics that could potentially tip off the attacker to your investigation. Until your investigation has achieved assurance in actor eviction, we strongly recommend that you keep all incident-related comms isolated to enable you to have the element of surprise when taking remediation actions.

Initial one-on-one and group communications can be achieved through phone (PSTN) calling, conference bridges not connected to the corporate infrastructure, and end-to-end encrypted messaging solutions.
One way that many customers have established secure productivity and collaboration is to create a new Office 365 tenant which is completely isolated from the organization’s production tenant and create accounts only for the key personnel needed, and any incident response vendors or partners who need to be part of the response.
- Make sure to follow best practices for securing this tenant, especially administrative accounts and rights by default. The new tenant should be limited on Administrative rights along with no trusts with outside applications or vendors. If you need further assistance or want information on hardening Microsoft 365, you can review the guidance here.

Investigate your environment

Once your incident responders and key personnel have a secure place to collaborate, the next step is to investigate the suspected compromised environment. Successful investigation will be a balance between getting to the bottom of every anomalous behavior to fully scope the extent of attacker activity and persistence and taking action quickly to stop any further activity on objectives by the attacker. Successful remediation requires as complete an understanding of the initial method of entry and persistence mechanisms controlled by the attacker as possible. Any persistence mechanisms missed could result in continued access by the attacker and potential for re-compromise.

Investigate and review cloud environment logs for suspicious actions and attacker IOCs, including:
- Unified Audit Logs (UAL).
- Azure Active Directory (Azure AD) logs.
- Active Directory logs.
- Exchange on-prem logs.
- VPN logs.
- Engineering systems logging.
- Antivirus and endpoint detection logging.
Review endpoint audit logs for changes from on-premises for actions including, but not limited to, the following:
- Group membership changes.
- New user account creation.
- Delegations within Active Directory.
- Along with other typical signs of compromise or activity.
Review Administrative rights in your environments

You may find that one or more of the logging sources above are data sources that the organization does not currently include in its security program. Some of them, especially the logging available in the cloud, are available only if configured and we recommend that you configure them as soon as possible to enable both the detections in the next section and forensics review of logs going forward. Make sure to configure your log retention to support your organization’s investigation goals going forward and retain evidence, if needed for legal, regulatory, or insurance purposes.

Establish continuous monitoring

There are many ways to detect activity associated with this campaign. Exactly how your organization will detect attacker behavior depends on which security tools you have available, or choose to deploy in response. Microsoft has provided examples publicly for some of the core security products and services that we offer and are continually updating those documents as new threat intelligence is identified related to this attacker. If you use other vendor’s products, review your vendor’s recommendations, and review the Microsoft documentation below to understand the detection techniques if you need to implement analogous detections in your environment on your own.

For readers using Azure Sentinel in their environments, review SolarWinds Post-Compromise Hunting guidance.

For readers using Microsoft Defender for Endpoint, review our guidance here, and review Microsoft Defender Antivirus guidance.

To Learn More go here