Deep Dive on Access Denied error when I use an IP-restriced IAM policy to access SSE-KMS encrypted S3 object

Context and Purpose

This article is going to provide deep dive and reveal root cause which was not included in article:

How do I resolve Access Denied errors when I use an IP-restricted IAM role to access SSE-KMS encrypted S3 objects?

TL: DR — If you find this article and just want to resolve the issue with no interest to know why, you can just follow the re:Post article that I haved posted above.

Background

I was trying to update a file into my S3 bucket with customer-managed KMS key. I was using a IAM identity with Administration permission. For security reason, I give the IAM user a IP restriction:

. {
.   "Sid": "statement-1",
.   "Effect": "Deny",
.   "Action": [
.     "*"
.   ],
.   "Resource": [
.     "*"
.   ],
.   "Condition": {
.     "NotIpAddress": {
.       "aws:SourceIp": [
.         "1.2.3.4/32" < -  -  -  - - masked from the real IP
.         ]
.      }
.    }
. }

I tried to upload it from AWS CLI and I got 403 Access Denied.

"User: arn:aws:iam::<aws-account>:<IAM identity type>/<identity-name> 
is not authorized to perform: kms:GenerateDataKey on 
resource: arn:aws:kms:<region>:<aws-account>:key/<KMS-Key-ID> 
with an explicit deny in an identity-based policy"

I suspected it was related to the restriction IAM policy, so I removed and tested it again.

It succeed. I uploaded the file into an S3 bucket with KMS encryption.

But why it did not work with existence of the IAM IP restriction policy?

Deep Dive:

The article does not tell the reason. And I did not find the article when I was troubleshooting either. Otherwise, I might figure out the root cause sooner.

Back to my troubleshooting, I enabled CloudTrail data event for faster API log delivery for S3 uploading object API call “PutObject” that I made. I checked the CloudTrail logs and It showed my request “s3:PutObject” was from the permitted source IP with the right IAM identity.

I did not look for the “kms:GenerateDataKey” API call at the time because I thought that it would be S3 service which was making the request.

From S3 document here:

Permissions

To successfully make a PutObject request to encrypt an object 
with an AWS KMS key to Amazon S3, you need kms:GenerateDataKey permissions 
on the key.

My IAM identity was already having the permission, because it has Administrator permission.

Get Jason’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

What did I miss?

I missed this:

To audit the usage of your AWS KMS keys for your SSE-KMS encrypted data, 
you can use AWS CloudTrail logs. You can get insight into your cryptographic 
operations, such as GenerateDataKey and Decrypt. 

CloudTrail supports numerous attribute values for filtering your search, 
including event name, user name, and event source.

I will show why it is important. I found this in CloudTrail Event History which is enabled by default in AWS account.

. "eventVersion": "1.11",
. "userIdentity": {
. "type": "<IAM Identity Type>",
. "principalId": "<masked>",
. "arn": "arn:aws:iam::<aws-account>:<IAM identity type>/<identity-name>",
. "accountId": "<aws-account>",
. "accessKeyId": "<masked>",
. "userName": "<masked>",
. "sessionContext": {
. "attributes": {
.   "creationDate": "<masked>",
.   "mfaAuthenticated": "<masked>"
. }
. },
. "invokedBy": "AWS Internal"
. },
. "eventTime": "<masked>",
. "eventSource": "kms.amazonaws.com",
. "eventName": "GenerateDataKey",
. "awsRegion": "<masked>",
. "sourceIPAddress": "AWS Internal",
. "userAgent": "AWS Internal",
. "errorCode": "AccessDenied",
. "errorMessage":"User: 
    arn:aws:iam::<aws-account>:<IAM identity type>/<identity-name> 
    is not authorized to perform: kms:GenerateDataKey on resource: 
    arn:aws:kms:us-east-1:<aws-account>:key/<KMS-Key-ID> with an explicit deny 
    in an identity-based policy",

truncated the rest

What do I learn from this Trail Event:

1. The requester making “kms: GenerateDataKey” API call is still treated as my IAM identity.
2. The API call was made by S3 service, but the API event showed it as “AWS Internal”.
3. The most important part, the “sourceIPAddress” element is marked as “AWS Internal”.

The 3rd point was the exact reason why the S3 uploading request failed all the time. The source IP passed for check with IAM policy was never going to pass because it was not even in an IP address format.

Condition Key that I tested

I tested the following condition keys which worked for me. I leave their details as further read for viewers of this article.

1. “aws:CalledVia” :

"ForAnyValue:StringEquals": {
     "aws:CalledVia": [
      "s3.amazonaws.com"
     ]
    }

1. 2. “aws:CalledViaFirst”:

"ForAnyValue:StringEquals": {
     "aws:CalledViaFirst": [
      "s3.amazonaws.com"
     ]
    }

3. “aws:ViaAWSService”:

"Bool": {
 "aws:PrincipalIsAWSService": "false",
 "aws:ViaAWSService": "false"
}

Conclusion: What do I learn from this:

1. CloudTrail Trail Event did help, because it delivers several times in one hour, often ever 5 minutes.
2. 2. I can still find the API call made by my requesting IAM identity but via an AWS service in CloudTrail Event History.