Tuesday, 4 October 2011

AWS CloudFront Secure Streaming


This is a followup to an answer I wrote on stackoverflow a few months ago about how to set up signed URLs with AWS CloudFront private streaming.

At the time, the python boto library had limited support for signed URLs and some of the steps were fairly hacky.  Since then, I've submitted some code to boto which will make secure signed URLs much easier. I've rewritten my answer here, making use of the new code. This code requires the 2.1 version of boto. Once version 2.1 of boto is commonly released I will update the stackoverflow answer as well.

To set up secure private CloudFront streaming with signed URLs you need to perform the following steps which I will detail below:
  1. Connect, create your s3 bucket, and upload some objects
  2. Create a Cloudfront "Origin Access Identity" (basically an AWS account to allow cloudfront to access your s3 bucket)
  3. Modify the ACLs on your private objects so that only your Cloudfront Origin Access Identity is allowed to read them (this prevents people from bypassing Cloudfront and going direct to s3)
  4. Create a cloudfront distribution that requires signed URLs
  5. Test that you can't download private object urls from s3 or the signed cloudfront distribution
  6. Create a key pair for signing private URLs
  7. Generate some private URLs using Python
  8. Test that the signed URLs work
Each step show a code snippet to perform that step.  All the snippets are combined into a single script for reference at the end.
1 - Connect, Create Bucket, and upload object
The easiest way to upload private objects is through the AWS Console but for completeness I'll show how using boto.  Boto code is shown here:
import boto

#credentials stored in environment AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
s3 = boto.connect_s3()
cf = boto.connect_cloudfront()

#bucket name MUST follow dns guidelines
new_bucket_name = "stream.example.com"
bucket = s3.create_bucket(new_bucket_name)

object_name = "video.mp4"
key = bucket.new_key(object_name)
key.set_contents_from_filename(object_name)

2 - Create a Cloudfront "Origin Access Identity"
This identity can be reused for many different distributions and keypairs. It is only used to allow cloudfront to access your private S3 objects without allowing everyone. As of now, this step can only be performed using the API. Boto code is here:
# Create a new Origin Access Identity
oai = cf.create_origin_access_identity(comment='New identity for secure videos')

print("Origin Access Identity ID: %s" % oai.id)
print("Origin Access Identity S3CanonicalUserId: %s" % oai.s3_user_id)

3 - Modify the ACLs on your objects
Now that we've got our special S3 user account (the S3CanonicalUserId we created above) we need to give it access to our private s3 objects. We can do this easily using the AWS Console by opening the object's (not the bucket's!) Permissions tab, click the "Add more permissions" button, and paste the very long S3CanonicalUserId we got above into the "Grantee" field of a new permission.  Make sure you give the new permission "Open/Download" rights.

You can also do this in code using the following boto script:
# Add read permission to our new s3 account
key.add_user_grant("READ", oai.s3_user_id)

4 - Create a cloudfront distribution
Note that custom origins and private distributions are only recently supported in boto version 2.1. To use these instructions you must get the latest release.

There are two important points here:
First, we are specifying an origin with an Origin Access Identifier. This allows CloudFront to access our private S3 objects without making the S3 bucket public. Users must use CloudFront to access the content.

The Second, is that we are specifying a "trusted_signers" parameter of "Self" to the distribution. This is what tells CloudFront that we want to require signed URLs. "Self" means that we will accept signatures from any CloudFront keypair in our own account. You can also give signing rights to other accounts if you want to allow others to create signed URLs for the content.
# Create an Origin object for boto
from boto.cloudfront.origin import S3Origin
origin = S3Origin("%s.s3.amazonaws.com" % new_bucket_name, oai)

# Create the signed distribution
dist = cf.create_distribution(origin=origin, enabled=True,
                              trusted_signers=["Self"],
                              comment="New distribution with signed URLs")

# Or, create a signed streaming distribution
stream_dist = cf.create_streaming_distribution(origin=origin, enabled=True,
                              trusted_signers=["Self"],
                              comment="New streaming distribution with signed URLs")

5 - Test that you can't download unsigned urls from cloudfront or s3
You should now be able to verify:

- stream.example.com.s3.amazonaws.com/video.mp4 - should give AccessDenied
- signed_distribution.cloudfront.net/video.mp4 - should give MissingKey (because the URL is not signed)

6 - Create a keypair for CloudFront
I think the only way to do this is through Amazon's web site. Go into your AWS "Account" page and click on the "Security Credentials" link.  Click on the "Key Pairs" tab then click "Create a New Key Pair". This will generate a new key pair for you and automatically download a private key file (pk-xxxxxxxxx.pem). Keep the key file safe and private. Also note down the "Key Pair ID" from amazon as we will need it in the next step.

7 - Generate some URLs in Python
In order to generate signed CloudFront URLs with boto, you must have the M2Crypto python library installed.  If it is not installed the following commands will raise a NotImplementedError.

For a non-streaming distribution, you must use the full cloudfront URL as the resource, however for streaming we only use the object name of the video file.

#Set parameters for URL
key_pair_id = "APKAIAZCZRKVIO4BQ" #from the AWS accounts page
priv_key_file = "cloudfront-pk.pem" #your private keypair file
expires = int(time.time()) + 300 #5 min

# For a downloading (normal http) use the full name
http_resource = 'http://%s/video.mp4' % dist.domain_name # your resource
# Create the signed URL
http_signed_url = dist.create_signed_url(http_resource, key_pair_id, expires, private_key_file=priv_key_file)

# For a streaming (rtmp) distribution use just the base filename
stream_resource = "video"
# Create the signed URL
stream_signed_url = stream_dist.create_signed_url(stream_resource, key_pair_id, expires, private_key_file=priv_key_file)

# Some flash players don't like query params so we have to escape them
def encode_query_param(resource):
    enc = resource
    enc = enc.replace('?', '%3F')
    enc = enc.replace('=', '%3D')
    enc = enc.replace('&', '%26')
    return enc

stream_signed_url = encode_query_param(stream_signed_url)

print("Download URL: %s" % http_signed_url)
print("Streaming URL: %s" % stream_signed_url)

8 - Try out the URLs

Hopefully your streaming url should look something like this:

video%3FExpires%3D1309979985%26Signature%3DMUNF7pw1689FhMeSN6JzQmWNVxcaIE9mk1x~KOudJky7anTuX0oAgL~1GW-ON6Zh5NFLBoocX3fUhmC9FusAHtJUzWyJVZLzYT9iLyoyfWMsm2ylCDBqpy5IynFbi8CUajd~CjYdxZBWpxTsPO3yIFNJI~R2AFpWx8qp3fs38Yw_%26Key-Pair-Id%3DAPKAIAZRKVIO4BQ

Put this into your js and you sould have something which looks like this:
var so_canned = new SWFObject('http://location.domname.com/~jvngkhow/player.swf','mpl','640','360','9');
so_canned.addParam('allowfullscreen','true');
so_canned.addParam('allowscriptaccess','always');
so_canned.addParam('wmode','opaque');
so_canned.addVariable('file','video%3FExpires%3D1309979985%26Signature%3DMUNF7pw1689FhMeSN6JzQmWNVxcaIE9mk1x~KOudJky7anTuX0oAgL~1GW-ON6Zh5NFLBoocX3fUhmC9FusAHtJUzWyJVZLzYT9iLyoyfWMsm2ylCDBqpy5IynFbi8CUajd~CjYdxZBWpxTsPO3yIFNJI~R2AFpWx8qp3fs38Yw_%26Key-Pair-Id%3DAPKAIAZRKVIO4BQ');
so_canned.addVariable('streamer','rtmp://s3nzpoyjpct.cloudfront.net/cfx/st');
so_canned.write('canned');

Summary
Post a comment if you have any trouble.

Enjoy!

86 comments:

  1. Hello Mike,
    A well written post, so a thanks for that.

    However there are a questions I did like to raise

    I installed boto from the present git (git clone). But when I try create_distribution, it gives me a error


    Traceback (most recent call last):
    File "", line 1, in
    TypeError: create_distribution() got an unexpected keyword argument 'trusted_sig ners'

    Can you please let me know which version has this.

    with regards

    ReplyDelete
  2. Hi adi.r.shanbhag,

    You caught me, I released this post before my code has been merged to boto master.

    In the mean time, you can try my branch:

    https://github.com/secretmike/boto

    I'm working with the boto folks to get this in the main boto release. I'll update here once that happens.

    Thanks,

    Mike

    ReplyDelete
  3. Hello Mike,
    Thanks for that. Using your branch for boto, I could create the private distribution (progressive download).

    Please be kind enough to advice/comment on the following:

    A couple of notes before I start:
    a. Python version is 2.6 on Ubuntu 10.04.3 64 bit
    b. Our Origin S3 has a files under subdirectories like /holder/001/012/SomegreatAudio.mp4
    c. I have verified that each file under the S3 bucket has read permissions for OAI.

    1. The above code examples you have demonstrated, uses create on variable dist and later on uses it to create the signed urls.

    Use case: when I have a OAI and distribution already created. Is it proper to define the following and go ahead

    cf = boto.connect_cloudfront()
    oai = cf.get_origin_access_identity_info(access_id)
    dist = cf.get_distribution_info(distribution_id)

    b. I have M2Crypto installed already. But while trying to create the signed url as demonstrated by you, I get this error

    Traceback (most recent call last):
    File "generate_signed_urls.py", line 24, in
    http_signed_url = dist.create_signed_url(http_resource, key_pair_id, expires, private_key_file=priv_key_file)
    File "/usr/local/lib/python2.6/dist-packages/boto-2.0-py2.6.egg/boto/cloudfront/distribution.py", line 560, in create_signed_url
    private_key_string=private_key_string)
    File "/usr/local/lib/python2.6/dist-packages/boto-2.0-py2.6.egg/boto/cloudfront/distribution.py", line 600, in _create_signing_params
    signature = self._sign_string(policy, private_key_file, private_key_string)
    File "/usr/local/lib/python2.6/dist-packages/boto-2.0-py2.6.egg/boto/cloudfront/distribution.py", line 660, in _sign_string
    key = EVP.load_key(private_key_file)
    File "/usr/local/lib/python2.6/dist-packages/M2Crypto-0.21.1-py2.6-linux-i686.egg/M2Crypto/EVP.py", line 370, in load_key
    raise EVPError(Err.get_error())
    M2Crypto.EVP.EVPError: 3075000000:error:0906D066:PEM routines:PEM_read_bio:bad end line:pem_lib.c:749:

    c. Since I hit this issue, I tried using the signed URL creation as demonstrated by you in the post "http://stackoverflow.com/questions/6549787/getting-started-with-secure-aws-cloudfront-streaming-with-python/6590986#6590986 "

    This creates a URL, but when I try it on the browser, I get "AccessDeniedAccess denied"

    Thank you

    ReplyDelete
  4. Hmm.. strange I made a very detailed comment for you to look at .. but seems it never got posted :(

    ReplyDelete
  5. Hi adi.r.shanbhag,

    Your post got marked as spam for some reason. I've un-spammed it so you should see it now!

    a) Yes, dist = cf.get_distribution_info(distribution_id) should get your existing distribution.

    b) It seems like M2Crypto is unable to read your PEM private key file. The "PEM_read_bio:bad end line" seems like your key file might be missing the ending line?

    c) There might be an issue with the url signing with unicode strings. Look for the following line:

    key.sign_update(message)

    and change it to:

    key.sign_update(str(message))

    (note that I've just corrected this in the stackoverflow answer.)

    Thanks,

    Mike

    ReplyDelete
  6. Hello Mike,
    Thank you for everything it all works like a charm.
    I have tested the Signed URL as demonstrated on stackoverflow answer and also the signed URL as demonstrated on this post. Both work great. Infact the method on this post is a cleaner and elegant way

    Thanks

    ReplyDelete
  7. Glad to hear everything is working. Any other CloudFront or AWS stuff you'd like to see an article about?

    Thanks,

    Mike

    ReplyDelete
  8. In the last bit of code, I'm a little confused here:

    # # For a streaming (rtmp) distribution use just the base filename
    # stream_resource = "video"

    Should you really omit the file extension?

    ReplyDelete
  9. Hi Alex,

    This depends on your streaming player. Many players don't handle the extension. I tidied up the example player source code at the end of the article. That's an example from JWPlayer. Notice how the CloudFront domain is in the separate "streamer" variable. The "/cfx/st" in there is another thing required by flash that DOES NOT appear in your bucket path.

    Let me know how you get on - I may do another post just for streaming which covers the whole process.

    If anybody knows which player does or doesn't require the extension please comment here!

    ReplyDelete
  10. I've been trying to create a signed URL and haven't been successful yet. I'll do some more experimentation with various players (FlowPlayer, JWplayer) and post my results.

    Thanks for the great article.

    ReplyDelete
  11. Question:

    If you omit the file extension for JWPlayer and create a signed url based on that, when the JWPlayer requests the file as an mp3 wont it have the wrong signature?

    For example, imagine I have the following key in my S3 bucket:

    alex/mysong.mp3

    and I create a signed URL from the string "alex/mysong". When JWPlayer sends a request for "alex/mysong" cloudfront wouldn't be able to find that asset, because it doesn't exist.

    And if the player sends a request for "alex/mysong.mp3" the signature would match, because I signed "alex/mysong".

    It's likely I'm just unclear on how the RTMP protocol defines a request. I'll keep experimenting.

    ReplyDelete
  12. Another question: where are you telling the player that it should be looking for an mp4 file?

    ReplyDelete
  13. I finally figured out my problem by looking at the perl implementation that Amazon provides. It turns out that the 'expires' parameter is not optional when you're not using a canned policy.

    Anyway, thanks for the helpful article.

    ReplyDelete
  14. I am new to AWS, CloudFront, and boto and I was able to follow along your guide fairly easily. There was only one snag and that is the "key_pair_id". What is this exactly? Is it the "Key Pair Name" in the EC2 Network & Security Key Pairs panel or something different? Essentially I see no "Accounts" in my management console.

    Many thanks for the excellent documentation!

    ReplyDelete
  15. I get the following error on the create_distribution command. Anything I should do ?

    MalformedInput - Could not parse XML

    ReplyDelete
    Replies
    1. Hi Dominik,

      It's hard to tell from here. If you could, send me the full python traceback or better yet, make a very small python program that causes the problem and send me that. I can take a look.

      Thanks,

      Mike

      Delete
  16. This comment has been removed by the author.

    ReplyDelete
    Replies
    1. Hi Mike,

      Thank you for your code. I have tried this and on step 7 the console returns an error:

      # Create the signed URL
      stream_signed_url = stream_dist.create_signed_url(stream_resource, key_pair_id, expires, private_key_file=priv_key_file)

      # Error:
      File "/Library/Python/2.7/site-packages/M2Crypto/EVP.py", line 370, in load_key
      raise EVPError(Err.get_error())
      M2Crypto.EVP.EVPError: 140735097370976:error:0906D06C:PEM routines:PEM_read_bio:no start line:/SourceCache/OpenSSL098/OpenSSL098-44/src/crypto/pem/pem_lib.c:648:Expecting: ANY PRIVATE KEY

      Do you knoe what might cause the issue? Thanks in advance for your help.

      Delete
    2. Nevermind. The issue was related with an incorrect private key file. I generated a new cloudfront key and it works.

      Thanks you very much!

      Delete
  17. This comment has been removed by the author.

    ReplyDelete
  18. Hi Mike,
    Thank you for the detailed post. I am trying to use this approach to play videos on HTML5 player. Since HTML video player does not support RTMP, I have created a HTTP distribution. Once I print the HTTP Url -
    print("Download URL: %s" % http_signed_url)

    I am trying to use the printed video URL on the player manually (using chrome console and pasting the URL in video source attribute).

    But this just doesn't play, I am not sure what is wrong.

    Is it possible to use this for HTML5 video player with a Web distribution?

    ReplyDelete
  19. In addition, you will need some sort of an encoder software that will help you to transform the signal into a continuous stream. Watch Elementary free online

    ReplyDelete
  20. Knowing your foe is imperative in battling him adequately. Security ought to be educated by arrange barrier, as well as by utilizing the defenselessness of programming and systems utilized for malignant expectation. As PC assault devices and systems keep on advancing, we will probably observe real, life-affecting occasions sooner rather than later.hubstaff

    ReplyDelete
  21. Thanks for sharing this informative content with us. We are working on html5 video player . It is really helpful for me and I get my lots of solution.

    ReplyDelete
  22. The most well known kind of encoding in video streaming is streak streaming. The best preferred standpoint of blaze streaming is that it tends to be played back in any sort of a program.best iptv service 2019

    ReplyDelete
  23. NFL Sunday Ticket has been a popular television package for Direct TV which offers customers the chance to watch every football game, in or out of market, from their home.live streaming football

    ReplyDelete
  24. For instance, the term pixel, in advanced camera technology, implies picture component and is the main marker of how smooth the image will look when printed. bestsecurityplace.com

    ReplyDelete
  25. While you will increase some layer of privacy from having a chosen one officer, investor, executive, and so forth this privacy will be lost once the candidate is served a subpoena and approached to give the contact data to the proprietors of the organization. mejoresvpn

    ReplyDelete
  26. Thanks for sharing this helpful content with us. We are working on html5 video player and this article helped us alot.

    ReplyDelete
  27. While some websites may host videos that have been put together by amateurs, there are some that will only host videos that are more professionally made. moviebox app

    ReplyDelete
  28. "Summary
    Post a comment if you have any trouble." - No troubles, just wanted to say "thanks!"

    Remote office Backup

    ReplyDelete
  29. That is a great tip especially to those new to the blogosphere. PDPA officer

    ReplyDelete
  30. Hi, This is a nice article you shared great information i have read it thanks for giving such a wonderful Blog for the reader. tree removal service wellington

    ReplyDelete
  31. I have read your article, it is very informative and helpful for me.I admire the valuable information you offer in your articles. Thanks for posting it.
    tree removal hollywood fl

    ReplyDelete
  32. Great article and a nice way to promote online. I’m satisfied with the information that you provided tree removal services broward county

    ReplyDelete
  33. This is a great article thanks for sharing this informative information. I will visit your blog regularly for some latest post.commercial dumpster rental jackson

    ReplyDelete
  34. Thanks for the wonderful share. Your article has proved your hard work and experience you have got in this field. Brilliant .i love it reading. tree trimming services dayton

    ReplyDelete
  35. A superbly written article, if only all bloggers offered the same content as you, the internet would be a far better place. junk removal services fort wayne

    ReplyDelete
  36. You have a good point here!I totally agree with what you have said!! Thanks for sharing your views. hope more people will read this article!!! septic tank replacement fort wayne

    ReplyDelete
  37. I think this is one of the most significant information for me. And i’m glad reading your article. tree care companies des moines

    ReplyDelete
  38. This post is good enough to make somebody understand this amazing thing, and I’m sure everyone will appreciate this interesting things.office cleanout services fort collins

    ReplyDelete
  39. I have read your article, it is very informative and helpful for me.I admire the valuable information you offer in your articles. Thanks for posting it..porta potty rental fort collins

    ReplyDelete
  40. I think this is one of the most significant information for me. And i’m glad reading your article. commercial tree services fresno

    ReplyDelete
  41. Excellent Post! For more information Visit Here.dumpster rental for construction bakersfield

    ReplyDelete
  42. This is a great article thanks for sharing this informative information. I will visit your blog regularly for some latest post.commercial septic systems bakersfield

    ReplyDelete
  43. Hello there! I could have sworn I’ve visited this website before but after looking at a few of the articles I realized it’s new to me.news Anyways, I’m certainly delighted I found it and I’ll be book-marking it and checking back often!

    ReplyDelete
  44. This excellent website truly has all of the technology info I wanted concerning this subject and didn’t know who to ask.

    ReplyDelete
  45. Excellent webpage. Thanks for sharing such a useful post.Very informative article. I liked it. Thanks for sharing this quality information with us. I really enjoyed reading.
    angular js training in chennai

    angular js online training in chennai

    angular js training in bangalore

    angular js training in hyderabad

    angular js training in coimbatore

    angular js training

    angular js online training

    ReplyDelete
  46. https://www.truckonroute.ca/ thanks

    ReplyDelete
  47. AngularJS is a structural framework for dynamic web apps. It lets you use HTML as your template language and lets you extend HTML's syntax to express your application's components clearly and succinctly.
    tally training in chennai

    hadoop training in chennai

    sap training in chennai

    oracle training in chennai

    angular js training in chennai

    ReplyDelete
  48. This post is really amazing
    Village Talkies a top-quality professional corporate video production company in Bangalore and also best explainer video company in Bangalore & animation video makers in Bangalore, Chennai, India & Maryland, Baltimore, USA provides Corporate & Brand films, Promotional, Marketing videos & Training videos, Product demo videos, Employee videos, Product video explainers, eLearning videos, 2d Animation, 3d Animation, Motion Graphics, Whiteboard Explainer videos Client Testimonial Videos, Video Presentation and more for all start-ups, industries, and corporate companies. From scripting to corporate video production services, explainer & 3d, 2d animation video production , our solutions are customized to your budget, timeline, and to meet the company goals and objectives.
    As a best video production company in Bangalore, we produce quality and creative videos to our clients.

    ReplyDelete
  49. I like your blog very much. Thanks for sharing such amazing blogs always. Just Love You Hoodie

    ReplyDelete
  50. I like the way you express information to us. Thanks for such post and please keep it up. Ferris Bueller Leather Jacket

    ReplyDelete



  51. Really nice article and helpful me
    10ml bottle packaging

    ReplyDelete
  52. very interesting , good job and thanks for sharing such a good blog.
    birthday wishes for someone special

    ReplyDelete
  53. custom paper cone sleeve this website is just outstanding for custom packaging boxes check it out

    ReplyDelete
  54. Custom packaging boxes are a great way to show off your product. Whether you're selling a product or giving it as a gift, custom packaging boxes can help you make sure that your customers get exactly what they want.

    Custom packaging boxes are usually made of cardboard but can also be made out of other materials like styrofoam and plastic. These CBD Packaging Boxes can be designed in various shapes, sizes and colors so that they can match the style of your product. If you have a small business that sells products that don't need to be kept in their original packaging, then you may be able to save money by using these custom packaging boxes instead of the standard ones sold by retailers like Target and Walmart.

    ReplyDelete
  55. It offers the easiest way to convert single or multiple files at once. iDealshare VideoGo Serial Key 2022 with Crack helps you activate premium. iDealshare VideoGo Crack

    ReplyDelete
  56. Kaspersky Anti-Virus Crack is a background of four different output ports that are full, quick, custom, and removable drive-mode. Kaspersky Crack

    ReplyDelete
  57. A new day, be open enough to see the opportunities. Be wise enough to be grateful. Be courageous enough to be happy. Happy Thursday. https://wishesquotz.com/thursday-quotes-messages/

    ReplyDelete
  58. Excellent blog. This is good, but you can do better.
    Native Instruments Massive

    ReplyDelete
  59. Presumably, the age 21 restriction is due to the of} sale of alcohol in that location. If may have} risk factors for compulsive playing, think about avoiding playing in any type, individuals who gamble and places where playing happens. Get therapy at the earliest signal of a problem to help stop playing from changing into worse. It's a good idea to get as many 점보카지노 welcome bonus provides as you'll be able to|you presumably can}. You're not allowed to join an online playing website more than once as} in order to to} claim a number of} preliminary deposit bonus provides.

    ReplyDelete