In an increasingly digital society, enterprise systems and software services offer various solutions that address the needs of government entities, organizations, and small businesses. The inner workings of these systems and services rely on vital components such as API keys, access tokens, and secrets to deliver business functionality to their clients. An API (Application Programming Interface) allows software components to connect and communicate with one another. API keys are a unique series of characters that grant verified access to an API and keys can be obtained through the permission of the API owner. Access tokens are similar to API keys; however, they contain a limited scope of what can be accessed and have a temporary lifespan. Secrets are sensitive credentials or privileged information that are contained or used within an application. These components are often connected to systems or services that store sensitive or business-critical data, and the increased reliance on them incentivizes cybercriminals to conduct cyberattacks. We explore Microsoft’s investigative report of the Storm-0558 key acquisition, lessons learned, other incidents, and recommendations to secure API keys, access tokens, and secrets.
On July 11, 2023, Microsoft published an initial post of a cyberattack involving the advanced persistent threat (APT) actor, tracked as Storm-0558, accessed and exfiltrated unclassified email data from various government agencies. The threat actor gained access to enterprise email accounts on Outlook Web Access in Exchange Online (OWA) and Outlook.com by discovering a leaked Microsoft Account (MSA) Consumer key, which enabled the threat actor to forge access tokens to the enterprise email accounts. MSA Consumer Keys allow a user to cryptographically sign into a Microsoft consumer service, while an access token is a string that enables clients to call protected web APIs securely.
Microsoft’s Investigative Report of Storm-0558 Key Acquisition
On September 6, 2023, Microsoft published the results of their investigative report on how Storm-0558 acquired the MSA Consumer Key used to forge access tokens to OWA and Outlook.com. A consumer signing system crash in April 2021 led to a snapshot of the crashed process to be stored in a “crash dump.” Crash dumps are created when an application faces an exception/error when running its code. These crash dumps contain vital diagnostic data that assist a software development team in understanding what caused the error. As per standard Microsoft debugging procedure, the crash dump should have been cleaned of any sensitive data, such as the signing keys or access tokens, before being moved into a debugging environment. However, Microsoft’s credential scan failed to detect sensitive information in the crash dump. The APT actor retrieved the key when they compromised a Microsoft engineer’s corporate account, inadvertently giving the hackers access to their debugging environment. This debugging environment included the crash dump that contained the consumer key. However, to access enterprise applications, an enterprise key is needed. In September 2018, Microsoft introduced a common key metadata publishing endpoint that allows customers to access various accounts with a single click. To accommodate this change, Microsoft updated its documentation and libraries to automatically check the scope of the keys. The scope of the keys determines whether a key is authorized to access a consumer or enterprise account. However, the libraries that perform this scope validation failed to verify the key type. Therefore, the mail system accepted access to an enterprise email using a consumer key that was then used to forge access tokens to OWA and Outlook.com.
Lessons Learned
The Storm-0558 key acquisition highlights that the Azure AD Software Development Kit (SDK) should have included better documentation for validating an access/authentication token’s issuer ID, which would have enabled developers both within Microsoft and outside the organization to better implement token authentication. Also, any debugging logs and crash dumps that store secrets should be disposed of routinely or when no longer needed. Additionally, mechanisms that scan components for secrets should be regularly tested and monitored to ensure their efficacy. Furthermore, keys and tokens should be rotated or set to expire regularly to avoid any potential or negative impacts of a breach of API keys or access tokens.
Other Incidents
Earlier this year, on February 7, 2023, the Cybernews research team discovered publicly accessible environment files hosted on Lowe’s Market website that leaked access tokens to AWS S3 buckets containing website-related assets and API keys to third-party services. These API keys provide access to various website and partner software functionality and may have allowed threat actors to steal user information, access partial credit card information, change product pricing, use the company’s official communication channels, and send emails to Lowe’s Market users.
On August 30, 2023, Sourcegraph, an AI-assisted coding platform, confirmed a security breach that led to the access of limited data, such as the license key holder’s name and email addresses for paid customers and account email addresses for community users. Malicious actors gained access to Sourcegraph’s data through a leaked administrative access token that was accidentally pushed to their code repository by a Sourcegraph engineer. Using the administrative access token, the threat actor created a new account with elevated privileges that was later used to navigate their admin dashboard containing user information.
More recently, on September 23, OpenSea, a Non-Fungible Tokens (NFT) marketplace, notified their customers of a breach with a third-party vendor. The breach exposed the API keys of OpenSea’s customers. OpenSea attempted to mitigate the risks of the API leak by informing users that their current keys would expire on October 2, 2023 and that clients should replace the expired keys. Although OpenSea has placed rate limits on the usage of APIs per key, this incident highlights the cyber risks of trusted third-party vendors and their impact of breaches on organizations.
Recommendations
Although every business has its own unique business-critical infrastructure or software, a few basic principles can be applied to all business-critical infrastructure or system software:
- Any secrets, such as passwords, API keys, access tokens, or personally identifiable information (PII), should not be stored in plaintext within logging environments. Encrypt secrets or tokens.
- Implement an expiration or rotation schedule for API keys or access tokens.
- Identify failure points in generating, verifying, and accepting access tokens or API keys and automate the process of updating these points whenever a change has been made.
- Implement the Principle of Least Privilege for API keys or access tokens.
- Set up logging capabilities to track the usage of secrets within your systems or software services.