Okta Incident Response Notes and Threat Hunting in Okta
The biggest downside of using a shared service is that you are vulnerable to cyber incidents like this one. Okta has identified adversarial activity that leveraged access to a stolen credential to access Okta’s support case management system. The threat actor was able to view files uploaded by certain Okta customers as part of recent support casesTraditional architectures are designed according to the fact that the attack will always come from the internet or unknown sources and defense mechanisms are built accordingly. But this is where the real danger comes in and you can get attacked from the application you trust. Your castle has been conquered from the inside, and if you don’t have enough monitoring power, you will often find out later and intervene too late.
Anyway, I would like to briefly share how we responded to the incident during the process, what we learned, what were our mistakes and what we did right. For the case details,
What I experienced in the process as an information security incident response,
On the 5th the first suspicious okta alert was generated, it should be noted that okta shared incident details after two weeks around 20th of Oct.
- Attackers hijacked the session of admin user from new geo location after finding a valid session cookie in a HAR file, ( they got this from okta support management system ) giving them access to okta admin console.
- Attacker created an API token
Okta publishes this login event types system.api_token.create . So you can generate an alarm event with this one as suspicious request and you must verify the activity with the admin users. The reason of the token creation. Beside this it will be another good alert point is, in syslog event you will find system.client.concurrency_rate_limit.notification.
- Attacker listed the permission of the admin
- Enumerated org configuration, tried to build a list of users, groups and applications
- Checked identity providers, Sign-On policies, MFA configs , Users, Groups etc.
When there is such a security incident with the identity management system, it will be much more effective to approach it as if these things have been done for sure, and to try to determine the greatest view of the damage.
Although you automate your threat model, send a message to the administrator via soar and ask him to respond to the case, and define escalation steps, after a while, unfortunately, alarm blindness occurs because they receive similar alarms a lot. the administrator thinks that it is one of the activities he does constantly and does not care, soc analysts can evaluate it as the daily activity of the administrators. because the password change that the attacker applies next, trying to enter other applications inside with sso, application information, etc. many activities such as application information are among the daily operations. If you do not detect this activity with first two event types when it happens in the beginning and intervene quickly, unfortunately it may be too late for respond it properly. Another learned lesson.
During the hunting I would say these event types might be critical to monitor closely;
Event Types
user.account.privilege.grant
debugContext.debugData.privilegeGranted=”Super administrator” group.privilege.grant
policy.rule.update
policy.rule.delete
application.policy.sign_on
system.api_token.create
system.api_token.revoke
security.threat.configuration.update
zone.delete
zone.deactivate
user.account.lock
“security.threat.detected” “outcome.reason”=”Password Spray”
user.account.report_suspicious_activity_by_enduser
user.mfa.factor.deactivate
user.mfa.factor.deactivate
user.mfa.factor.reset_all
system.email.mfa_reset_notification.sent_message
Important for the debug data, it will provide new ip new location new device details. It will give fresh IP addresses, locations, and device details, all of which are crucial for the debug data. Keep a focused eye on admin login activity; if you see any unusual device behavior or location, for example, you can send out an alert.
For the threat hunting part, you must keep in mind that attacker will try to lateral movement activity, with SSO permission adversary group will try to jump other systems such as cloud management, vpn, code repo, password manager, email system etc. So if there is any changed username or group information just trace the same username activity in the all environments logs indexes etc.
Important note for these kinds of incidents: submit a high severity support issue with the vendor right away. It’s likely that you may receive a delayed response from them because they are currently conducting an investigation and need time to analyze the information. Additionally, they have other activities ongoing on their end. Thus, in order to defend against these dangerous supply chain threats, you need to be alert, quick, and organized. It’s likely that their assistance will arrive much later. ( 2 weeks in this okta case)
Lessons
- Keep a careful eye on super admin accounts. Whether using an otomated or SOC analyst, the procedure should still be efficient, well-organized, and quick.
- You are alone with the incident until clarification comes from the vendor till that time you have to cover all investigation and share your finding with the investigation team. Response is the critical part of the cyber incidents, after the proper response you will have tons of times for the detailed investigation.
- Critical event types put relevant team lead and managers also, admin may ignore it but someone else may spot. Admin might be busy but team lead can handle it. This process must be automatic for the big environments.
- Do not rely too much on outsourced services. the easiest entry points for attackers are 3party applications like this. You need to keep your threat modeling and alarms ready and monitor them, you don’t need to care about system administrators, they usually realize the criticality of the situation after the incident occurs, not everyone may approach the processes with security priority.
IOCs
