Tuesday, October 4, 2011

AWS CloudFront Secure Streaming


This is a followup to an answer I wrote on stackoverflow a few months ago about how to set up signed URLs with AWS CloudFront private streaming.

At the time, the python boto library had limited support for signed URLs and some of the steps were fairly hacky.  Since then, I've submitted some code to boto which will make secure signed URLs much easier. I've rewritten my answer here, making use of the new code. This code requires the 2.1 version of boto. Once version 2.1 of boto is commonly released I will update the stackoverflow answer as well.

To set up secure private CloudFront streaming with signed URLs you need to perform the following steps which I will detail below:
  1. Connect, create your s3 bucket, and upload some objects
  2. Create a Cloudfront "Origin Access Identity" (basically an AWS account to allow cloudfront to access your s3 bucket)
  3. Modify the ACLs on your private objects so that only your Cloudfront Origin Access Identity is allowed to read them (this prevents people from bypassing Cloudfront and going direct to s3)
  4. Create a cloudfront distribution that requires signed URLs
  5. Test that you can't download private object urls from s3 or the signed cloudfront distribution
  6. Create a key pair for signing private URLs
  7. Generate some private URLs using Python
  8. Test that the signed URLs work
Each step show a code snippet to perform that step.  All the snippets are combined into a single script for reference at the end.
1 - Connect, Create Bucket, and upload object
The easiest way to upload private objects is through the AWS Console but for completeness I'll show how using boto.  Boto code is shown here:
import boto

#credentials stored in environment AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
s3 = boto.connect_s3()
cf = boto.connect_cloudfront()

#bucket name MUST follow dns guidelines
new_bucket_name = "stream.example.com"
bucket = s3.create_bucket(new_bucket_name)

object_name = "video.mp4"
key = bucket.new_key(object_name)
key.set_contents_from_filename(object_name)

2 - Create a Cloudfront "Origin Access Identity"
This identity can be reused for many different distributions and keypairs. It is only used to allow cloudfront to access your private S3 objects without allowing everyone. As of now, this step can only be performed using the API. Boto code is here:
# Create a new Origin Access Identity
oai = cf.create_origin_access_identity(comment='New identity for secure videos')

print("Origin Access Identity ID: %s" % oai.id)
print("Origin Access Identity S3CanonicalUserId: %s" % oai.s3_user_id)

3 - Modify the ACLs on your objects
Now that we've got our special S3 user account (the S3CanonicalUserId we created above) we need to give it access to our private s3 objects. We can do this easily using the AWS Console by opening the object's (not the bucket's!) Permissions tab, click the "Add more permissions" button, and paste the very long S3CanonicalUserId we got above into the "Grantee" field of a new permission.  Make sure you give the new permission "Open/Download" rights.

You can also do this in code using the following boto script:
# Add read permission to our new s3 account
key.add_user_grant("READ", oai.s3_user_id)

4 - Create a cloudfront distribution
Note that custom origins and private distributions are only recently supported in boto version 2.1. To use these instructions you must get the latest release.

There are two important points here:
First, we are specifying an origin with an Origin Access Identifier. This allows CloudFront to access our private S3 objects without making the S3 bucket public. Users must use CloudFront to access the content.

The Second, is that we are specifying a "trusted_signers" parameter of "Self" to the distribution. This is what tells CloudFront that we want to require signed URLs. "Self" means that we will accept signatures from any CloudFront keypair in our own account. You can also give signing rights to other accounts if you want to allow others to create signed URLs for the content.
# Create an Origin object for boto
from boto.cloudfront.origin import S3Origin
origin = S3Origin("%s.s3.amazonaws.com" % new_bucket_name, oai)

# Create the signed distribution
dist = cf.create_distribution(origin=origin, enabled=True,
                              trusted_signers=["Self"],
                              comment="New distribution with signed URLs")

# Or, create a signed streaming distribution
stream_dist = cf.create_streaming_distribution(origin=origin, enabled=True,
                              trusted_signers=["Self"],
                              comment="New streaming distribution with signed URLs")

5 - Test that you can't download unsigned urls from cloudfront or s3
You should now be able to verify:

- stream.example.com.s3.amazonaws.com/video.mp4 - should give AccessDenied
- signed_distribution.cloudfront.net/video.mp4 - should give MissingKey (because the URL is not signed)

6 - Create a keypair for CloudFront
I think the only way to do this is through Amazon's web site. Go into your AWS "Account" page and click on the "Security Credentials" link.  Click on the "Key Pairs" tab then click "Create a New Key Pair". This will generate a new key pair for you and automatically download a private key file (pk-xxxxxxxxx.pem). Keep the key file safe and private. Also note down the "Key Pair ID" from amazon as we will need it in the next step.

7 - Generate some URLs in Python
In order to generate signed CloudFront URLs with boto, you must have the M2Crypto python library installed.  If it is not installed the following commands will raise a NotImplementedError.

For a non-streaming distribution, you must use the full cloudfront URL as the resource, however for streaming we only use the object name of the video file.

#Set parameters for URL
key_pair_id = "APKAIAZCZRKVIO4BQ" #from the AWS accounts page
priv_key_file = "cloudfront-pk.pem" #your private keypair file
expires = int(time.time()) + 300 #5 min

# For a downloading (normal http) use the full name
http_resource = 'http://%s/video.mp4' % dist.domain_name # your resource
# Create the signed URL
http_signed_url = dist.create_signed_url(http_resource, key_pair_id, expires, private_key_file=priv_key_file)

# For a streaming (rtmp) distribution use just the base filename
stream_resource = "video"
# Create the signed URL
stream_signed_url = stream_dist.create_signed_url(stream_resource, key_pair_id, expires, private_key_file=priv_key_file)

# Some flash players don't like query params so we have to escape them
def encode_query_param(resource):
    enc = resource
    enc = enc.replace('?', '%3F')
    enc = enc.replace('=', '%3D')
    enc = enc.replace('&', '%26')
    return enc

stream_signed_url = encode_query_param(stream_signed_url)

print("Download URL: %s" % http_signed_url)
print("Streaming URL: %s" % stream_signed_url)

8 - Try out the URLs

Hopefully your streaming url should look something like this:

video%3FExpires%3D1309979985%26Signature%3DMUNF7pw1689FhMeSN6JzQmWNVxcaIE9mk1x~KOudJky7anTuX0oAgL~1GW-ON6Zh5NFLBoocX3fUhmC9FusAHtJUzWyJVZLzYT9iLyoyfWMsm2ylCDBqpy5IynFbi8CUajd~CjYdxZBWpxTsPO3yIFNJI~R2AFpWx8qp3fs38Yw_%26Key-Pair-Id%3DAPKAIAZRKVIO4BQ

Put this into your js and you sould have something which looks like this:
var so_canned = new SWFObject('http://location.domname.com/~jvngkhow/player.swf','mpl','640','360','9');
so_canned.addParam('allowfullscreen','true');
so_canned.addParam('allowscriptaccess','always');
so_canned.addParam('wmode','opaque');
so_canned.addVariable('file','video%3FExpires%3D1309979985%26Signature%3DMUNF7pw1689FhMeSN6JzQmWNVxcaIE9mk1x~KOudJky7anTuX0oAgL~1GW-ON6Zh5NFLBoocX3fUhmC9FusAHtJUzWyJVZLzYT9iLyoyfWMsm2ylCDBqpy5IynFbi8CUajd~CjYdxZBWpxTsPO3yIFNJI~R2AFpWx8qp3fs38Yw_%26Key-Pair-Id%3DAPKAIAZRKVIO4BQ');
so_canned.addVariable('streamer','rtmp://s3nzpoyjpct.cloudfront.net/cfx/st');
so_canned.write('canned');

Summary
Post a comment if you have any trouble.

Enjoy!

26 comments:

  1. Hello Mike,
    A well written post, so a thanks for that.

    However there are a questions I did like to raise

    I installed boto from the present git (git clone). But when I try create_distribution, it gives me a error


    Traceback (most recent call last):
    File "", line 1, in
    TypeError: create_distribution() got an unexpected keyword argument 'trusted_sig ners'

    Can you please let me know which version has this.

    with regards

    ReplyDelete
  2. Hi adi.r.shanbhag,

    You caught me, I released this post before my code has been merged to boto master.

    In the mean time, you can try my branch:

    https://github.com/secretmike/boto

    I'm working with the boto folks to get this in the main boto release. I'll update here once that happens.

    Thanks,

    Mike

    ReplyDelete
  3. Hello Mike,
    Thanks for that. Using your branch for boto, I could create the private distribution (progressive download).

    Please be kind enough to advice/comment on the following:

    A couple of notes before I start:
    a. Python version is 2.6 on Ubuntu 10.04.3 64 bit
    b. Our Origin S3 has a files under subdirectories like /holder/001/012/SomegreatAudio.mp4
    c. I have verified that each file under the S3 bucket has read permissions for OAI.

    1. The above code examples you have demonstrated, uses create on variable dist and later on uses it to create the signed urls.

    Use case: when I have a OAI and distribution already created. Is it proper to define the following and go ahead

    cf = boto.connect_cloudfront()
    oai = cf.get_origin_access_identity_info(access_id)
    dist = cf.get_distribution_info(distribution_id)

    b. I have M2Crypto installed already. But while trying to create the signed url as demonstrated by you, I get this error

    Traceback (most recent call last):
    File "generate_signed_urls.py", line 24, in
    http_signed_url = dist.create_signed_url(http_resource, key_pair_id, expires, private_key_file=priv_key_file)
    File "/usr/local/lib/python2.6/dist-packages/boto-2.0-py2.6.egg/boto/cloudfront/distribution.py", line 560, in create_signed_url
    private_key_string=private_key_string)
    File "/usr/local/lib/python2.6/dist-packages/boto-2.0-py2.6.egg/boto/cloudfront/distribution.py", line 600, in _create_signing_params
    signature = self._sign_string(policy, private_key_file, private_key_string)
    File "/usr/local/lib/python2.6/dist-packages/boto-2.0-py2.6.egg/boto/cloudfront/distribution.py", line 660, in _sign_string
    key = EVP.load_key(private_key_file)
    File "/usr/local/lib/python2.6/dist-packages/M2Crypto-0.21.1-py2.6-linux-i686.egg/M2Crypto/EVP.py", line 370, in load_key
    raise EVPError(Err.get_error())
    M2Crypto.EVP.EVPError: 3075000000:error:0906D066:PEM routines:PEM_read_bio:bad end line:pem_lib.c:749:

    c. Since I hit this issue, I tried using the signed URL creation as demonstrated by you in the post "http://stackoverflow.com/questions/6549787/getting-started-with-secure-aws-cloudfront-streaming-with-python/6590986#6590986 "

    This creates a URL, but when I try it on the browser, I get "AccessDeniedAccess denied"

    Thank you

    ReplyDelete
  4. Hmm.. strange I made a very detailed comment for you to look at .. but seems it never got posted :(

    ReplyDelete
  5. Hi adi.r.shanbhag,

    Your post got marked as spam for some reason. I've un-spammed it so you should see it now!

    a) Yes, dist = cf.get_distribution_info(distribution_id) should get your existing distribution.

    b) It seems like M2Crypto is unable to read your PEM private key file. The "PEM_read_bio:bad end line" seems like your key file might be missing the ending line?

    c) There might be an issue with the url signing with unicode strings. Look for the following line:

    key.sign_update(message)

    and change it to:

    key.sign_update(str(message))

    (note that I've just corrected this in the stackoverflow answer.)

    Thanks,

    Mike

    ReplyDelete
  6. Hello Mike,
    Thank you for everything it all works like a charm.
    I have tested the Signed URL as demonstrated on stackoverflow answer and also the signed URL as demonstrated on this post. Both work great. Infact the method on this post is a cleaner and elegant way

    Thanks

    ReplyDelete
  7. Glad to hear everything is working. Any other CloudFront or AWS stuff you'd like to see an article about?

    Thanks,

    Mike

    ReplyDelete
  8. In the last bit of code, I'm a little confused here:

    # # For a streaming (rtmp) distribution use just the base filename
    # stream_resource = "video"

    Should you really omit the file extension?

    ReplyDelete
  9. Hi Alex,

    This depends on your streaming player. Many players don't handle the extension. I tidied up the example player source code at the end of the article. That's an example from JWPlayer. Notice how the CloudFront domain is in the separate "streamer" variable. The "/cfx/st" in there is another thing required by flash that DOES NOT appear in your bucket path.

    Let me know how you get on - I may do another post just for streaming which covers the whole process.

    If anybody knows which player does or doesn't require the extension please comment here!

    ReplyDelete
  10. I've been trying to create a signed URL and haven't been successful yet. I'll do some more experimentation with various players (FlowPlayer, JWplayer) and post my results.

    Thanks for the great article.

    ReplyDelete
  11. Question:

    If you omit the file extension for JWPlayer and create a signed url based on that, when the JWPlayer requests the file as an mp3 wont it have the wrong signature?

    For example, imagine I have the following key in my S3 bucket:

    alex/mysong.mp3

    and I create a signed URL from the string "alex/mysong". When JWPlayer sends a request for "alex/mysong" cloudfront wouldn't be able to find that asset, because it doesn't exist.

    And if the player sends a request for "alex/mysong.mp3" the signature would match, because I signed "alex/mysong".

    It's likely I'm just unclear on how the RTMP protocol defines a request. I'll keep experimenting.

    ReplyDelete
  12. Another question: where are you telling the player that it should be looking for an mp4 file?

    ReplyDelete
  13. I finally figured out my problem by looking at the perl implementation that Amazon provides. It turns out that the 'expires' parameter is not optional when you're not using a canned policy.

    Anyway, thanks for the helpful article.

    ReplyDelete
  14. I am new to AWS, CloudFront, and boto and I was able to follow along your guide fairly easily. There was only one snag and that is the "key_pair_id". What is this exactly? Is it the "Key Pair Name" in the EC2 Network & Security Key Pairs panel or something different? Essentially I see no "Accounts" in my management console.

    Many thanks for the excellent documentation!

    ReplyDelete
  15. I get the following error on the create_distribution command. Anything I should do ?

    MalformedInput - Could not parse XML

    ReplyDelete
    Replies
    1. Hi Dominik,

      It's hard to tell from here. If you could, send me the full python traceback or better yet, make a very small python program that causes the problem and send me that. I can take a look.

      Thanks,

      Mike

      Delete
  16. This comment has been removed by a blog administrator.

    ReplyDelete
    Replies
    1. Hi Mike,

      Thank you for your code. I have tried this and on step 7 the console returns an error:

      # Create the signed URL
      stream_signed_url = stream_dist.create_signed_url(stream_resource, key_pair_id, expires, private_key_file=priv_key_file)

      # Error:
      File "/Library/Python/2.7/site-packages/M2Crypto/EVP.py", line 370, in load_key
      raise EVPError(Err.get_error())
      M2Crypto.EVP.EVPError: 140735097370976:error:0906D06C:PEM routines:PEM_read_bio:no start line:/SourceCache/OpenSSL098/OpenSSL098-44/src/crypto/pem/pem_lib.c:648:Expecting: ANY PRIVATE KEY

      Do you knoe what might cause the issue? Thanks in advance for your help.

      Delete
    2. Nevermind. The issue was related with an incorrect private key file. I generated a new cloudfront key and it works.

      Thanks you very much!

      Delete
  17. This comment has been removed by the author.

    ReplyDelete
  18. Hi Mike,
    Thank you for the detailed post. I am trying to use this approach to play videos on HTML5 player. Since HTML video player does not support RTMP, I have created a HTTP distribution. Once I print the HTTP Url -
    print("Download URL: %s" % http_signed_url)

    I am trying to use the printed video URL on the player manually (using chrome console and pasting the URL in video source attribute).

    But this just doesn't play, I am not sure what is wrong.

    Is it possible to use this for HTML5 video player with a Web distribution?

    ReplyDelete
  19. I admire the valuable information you offer in your articles. I will bookmark your blog and have my friends check up here often. I am quite sure they will learn lots of new stuff here than anybody else! Regards aws jobs in hyderabad.

    ReplyDelete
  20. Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a Front end developer learn from Javascript Training in Chennai . or Javascript Training in Chennai. Nowadays JavaScript has tons of job opportunities on various vertical industry. ES6 Training in Chennai

    ReplyDelete
  21. In addition, you will need some sort of an encoder software that will help you to transform the signal into a continuous stream. Watch Elementary free online

    ReplyDelete
  22. Knowing your foe is imperative in battling him adequately. Security ought to be educated by arrange barrier, as well as by utilizing the defenselessness of programming and systems utilized for malignant expectation. As PC assault devices and systems keep on advancing, we will probably observe real, life-affecting occasions sooner rather than later.hubstaff

    ReplyDelete
  23. Thanks for sharing this informative content with us. We are working on html5 video player . It is really helpful for me and I get my lots of solution.

    ReplyDelete