Unable to access S3 file with IAM role from EC2

There are three things to note:

• How credentials are provided and accessed from an Amazon EC2 instance
• How to assign permissions for access to Amazon S3
• How Pre-Signed URLs function

1. How credentials are provided and accessed from an Amazon EC2 instance

When an Amazon EC2 instance is launched with an IAM Role, the Instance Metadata automatically provides temporary access credentials consisting of an Access Key, Secret Key and Token. These credentials are rotated approximately every six hours.

Any code that uses an AWS SDK (eg Python, Java, PHP) knows how to automatically retrieve these credentials. Therefore, code running on an Amazon EC2 instance that was launched with an IAM role does not require you to retrieve nor provide access credentials -- it just works automagically!

So, in your above code sample, you could remove any lines that specifically refer to credentials. Your job is simply to ensure that the IAM Role has sufficient permissions for the operations you wish to perform.

This also applies to the AWS Command-Line Interface (CLI), which is actually just a Python program that provides command-line access to AWS API calls. Since it uses the AWS SDK for Python, it automatically retrieves the credentials from Instance Metadata and does not require credentials when used from an Amazon EC2 instance that was launched with an IAM Role.

2. How to assign permissions for access to Amazon S3

Objects in Amazon S3 are private by default. There are three ways to assign permission to access objects:

• Object ACLs (Access Control Lists): These are permissions on the objects themselves
• Bucket Policies: This is a set of rules applied to the bucket as a whole, but it can also specify permissions related to a subset of a bucket (eg a particular path within the bucket)
• IAM Policies that are applied to IAM Users, Groups or Roles: These permissions apply specifically to those entities

Since you are wanting to grant access to Amazon S3 objects to a specific IAM User, it is better to assign permissions via an IAM Policy attached to that user, rather than being part of the Bucket Policy.

Therefore, you should:

• Remove the Bucket Policy
• Create an Inline Policy in IAM and attach it to the desired IAM User. The policy then applies to that User and does not require a Principal

Here is a sample policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::MY-BUCKET/*"
            ]
        }
    ]
}

I have recommended an Inline Policy because this policy applies to just one user. If you are assigning permissions to many users, it is recommended to attach the policy to an IAM Group and then the Users assigned to that group will in inherit the permissions. Alternatively, create an IAM Policy and then attach that policy to all relevant Users.

3. How Pre-Signed URLs function

Amazon S3 Pre-Signed URLs are a means of granting temporary access to Amazon S3 objects. The generated URL includes:

• The Access Key of an IAM User that has permission to access the object
• An expiration time
• A signature created via a has operation that authorises the URL

The key point to realise is related to the permissions used when generating the pre-signed URL. As mentioned in the Amazon S3 documentation Share an Object with Others:

Anyone with valid security credentials can create a pre-signed URL. However, in order to successfully access an object, the pre-signed URL must be created by someone who has permission to perform the operation that the pre-signed URL is based upon.

This means that the credentials used when generating the pre-signed URL are also the credentials used as part of the pre-signed URL. The entity associated with those credentials, of course, needs permission to access the object -- the pre-signed URL is merely a means of on-granting access to an object for a temporary period.

What this also means is that, in the case of your example, you do not need to create a specific role for granting access to the object(s) in Amazon S3. Instead, you can use a more permissive IAM Role with your Amazon EC2 instance (for example, one that can also upload objects to S3) but when it generates a pre-signed URL it is only granting temporary access to the object (and not other permissions, such as the upload permission).

If the software running on your Amazon EC2 instance only interacts with AWS to created signed URLs, then your Role that has only GetObject permissions is fine. However, if your instance wants to do more, then create a Role that grants the instance the appropriate permissions (including GetObject access to S3) and generate Signed URLs using that Role.

If you wish to practice generating signed URLs, recent versions of the AWS Command-Line Interface (CLI) includes a aws s3 presign s3://path command that can generate pre-signed URLs. Try with with various --profile settings to see how it works with different IAM Users.