My AWS Account is hacked! I am being billed for Services which I have not used!!! What can I do now?
Few months back I have posted my journey towards AWS Certified Solution Architect – Associate exam. The link can be found below,
http://ahirjoy.blogspot.com/2022/06/how-i-passed-aws-certified-architect.html
However, while my cloud journey was going on well, there was a rude shock when I realized that my account got hacked. Though I was on long leave last month, I observed that my billing was increasing exponentially everyday though I was sure that I did not have any services running. Still I logged in to my AWS account to verify the same as I started getting unusual usage alert from AWS support also by then. To my surprise, I found many EC2 instances, VPCs and policies were created for my account which were very much active! Also, I found one subscription NVIDIA GPU Optimized AMI was subscribed for me in AWS marketplace!!!
I got panicked. I Immediately closed the account. However, to my disappointment I received communication from AWS support that unless the account is secured, it can’t be closed cleanly and services may be still in use and I would be billed based on that continuously! (Strange?)
So I contacted few kind CloudGurus in my Circle and AWS support who were kind enough to guide me through the entire process of securing my AWS account. I am so much thankful to all of them and I am consolidating the steps below in great detail hoping it may help someone else in the time of need one day!
Step 1: Change your AWS root account password immediately.
See the root account password change guide: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_passwords_change-root.html .
Also, enable Multi Factor Authentication (MFA) without second thought!
As a further precaution, it is recommended to change your email password and passwords for other websites to help protect your account from being compromised again.
Step 2: Check your CloudTrail log for unsanctioned activity such as creation of unauthorized IAM users, policies, roles or temporary security credentials. To secure your account please delete any unauthorized IAM users, roles, and policies, and revoke any temporary credentials. NOTE: You cannot revoke temporary credentials obtained via the root user.
For more information see: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_disable-perms.html#denying-access-to-credentials-creator .
1. To delete unauthorized IAM users, go here: https://console.aws.amazon.com/iam/home#users .
2. To delete unauthorized policies, go here: https://console.aws.amazon.com/iam/home#/policies .
3. To delete unauthorized roles, go here: https://console.aws.amazon.com/iam/home#/roles .
4. You can revoke temporary credentials by following the instructions here: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_disable-perms.html#denying-access-to-credentials-by-issue-time .
Temporary credentials can also be revoked by deleting the IAM User. NOTE: Deleting IAM users may impact production workloads and should be done carefully. Step 3.
Do the following to check and delete for unauthorized AWS usage: -
1. Check your CloudTrail logs - Check your “Bills” page: https://console.aws.amazon.com/billing/home#/bill –
2. Check Cost Explorer: https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/ce-what-is.html
Some common usage types to check for are EC2 instances, EC2 Spot bids, Lambda functions, AMIs, EBS volumes, EBS snapshots, Lightsail instances, and Sagemaker notebook instances. For help with deleting resources associated with these services, see the following:
3. EC2 instances:
https://aws.amazon.com/premiumsupport/knowledge-center/delete-terminate-ec2/
4. EC2 Spot bids:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html#terminating-a-spot-instance
5. Lambda functions:
6. AMIs:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/deregister-ami.html
7. EBS volumes:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-deleting-volume.html
8. EBS snapshots:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-deleting-snapshot.html#ebs-delete-snapshot
9. Lightsail instances: https://lightsail.aws.amazon.com/ls/docs/en_us/articles/delete-an-amazon-lightsail-instance
10. Sagemaker notebook instances: https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-cleanup.html
Keep in mind that unauthorized usage can occur in any AWS Region and that your console may show you only one Region at a time. To switch between Regions, you can use the dropdown in the upper-right corner of the console screen.
Please note that for security reasons, AWS Support cannot terminate customer resources on your behalf, you need to follow the above steps as specified in the above links/documentation. but please follow these steps to delete any unauthorized resources:
Delete VPCs:
1. Select the security groups that are not marked as 'Yes' under "Default VPC"
2. Select in 'Actions'
3. Select 'Delete VPC'.
4. Change the region and repeat the steps above.
EC2 security groups
If unauthorized: Select the security groups that are not named as 'default' > Select in 'Actions' > Select 'Delete security groups'. ~ EC2 Key Pairs
To delete the Key Pair: 1. Log into the EC2 console
2. On the left panel choose Key Pairs.
3. Select the key pair you want to delete.
4. Select Actions > Delete.
With regards to the FES and Backup Roles :
Firstly, Please check if you have any backup vaults related to the Elastic file system service (EFS) here: If yes, please follow the steps provided in the documentation here to disable them: https://aws.amazon.com/premiumsupport/knowledge-center/efs-disable-automatic-backups/ ** The document also contains a video for reference. Once you clear the back up storage for EFS you should be able to delete the IAM roles related to the same. >>
To delete unauthorized roles, go here:
https://console.aws.amazon.com/iam/home#/roles If you are unable to find any backup vaults, please write back to us along with the screenshot of the same so that we can have our internal team investigate on it further.
To delete Root User Keys go here:
https://console.aws.amazon.com/iam/home#security_credential . If your application uses the exposed Access Key, you need to replace the Key. To replace the Key, first create a second Key (at that point both Keys will be active) and then modify your application to use the new Key. Then disable (but do not delete) the exposed Key by clicking on the “Make inactive” option in the console. If there are any problems with your application, you can reactivate the exposed Key. When your application is fully functional using the new Key, please delete the exposed Key. NOTE: Only rotating and deleting the exposed key may not be sufficient to protect your account, see Step 2. Step 2: Check your CloudTrail log for unsanctioned activity such as creation of unauthorized IAM users, access keys, login profiles, policies, roles or temporary security credentials. To secure your account please delete any unauthorized IAM users, access keys, login profiles, roles, and policies, and revoke any temporary credentials. NOTE: You cannot revoke temporary credentials obtained via the Root User. For more information see:
Finally you must set up at least two of the following services to monitor cost and usage: https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-create.html#create-cost-budget (** recommended)
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/gs_monitor_estimated_charges_with_cloudwatch.html#gs_creating_billing_alarm (** recommended)
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html
https://docs.aws.amazon.com/waf/latest/developerguide/getting-started.html
https://docs.aws.amazon.com/awssupport/latest/user/get-started-with-aws-trusted-advisor.html
For more information about managing your AWS cost and usage, see the following: https://docs.aws.amazon.com/cost-management/latest/userguide/what-is-costmanagement.html ======
2. You must set up at least one of the following security best practices: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa_enable_virtual.html#enable-virt-mfa-for-root (** recommended) https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-enable.html https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_settingup.html ======
3. Confirm that you have read the AWS Customer Agreement and Shared Responsibility Model: https://aws.amazon.com/agreement/ https://aws.amazon.com/compliance/shared-responsibility-model/ ======
After all these are done, you can request AWS Support for consideration of Billing Adjustment and hope for the best. But please note that even AWS considers your request once (as it was for me, I am really grateful for that), if you are careless with your account security, they will not consider again as securing one’s account is user’s responsibility. So please, please take the security measures very seriously.
Best of luck!
Comments
Post a Comment