Sharing #AWS S3 files for ansible thru Signed URLs

Hi Guys, I love automation. However, some much automation should get along with a secure way to share files like qcow2 images and rpm. I am doing that on my own and I would like to share what I’ve learnt
First of all, Thanks to Remi (@rvichery), Guillermo Alvarado (@galvarado89) and Boris Cortes (@boriscortes) for helping me out to find the best method here.

Prepare your system

It’d be great if you create a temporary user/group. My case I created a group called “read_only” like the next picture with the following attached permission “AmazonS3ReadOnlyAccess”

And then I created a user getting the AccessKeyID and the SecretKey. Install aws client. Guess what OS in my case. Nop! it isn’t Mac OSX now, I am doing that in my CentOS 🙂

yum -y install python-pip
pip install aws
pip install boto #checking if it was installed already

Configure your access with: “aws configure” and enter the required info (AccessKeyID and SecretKey)
Now, you can check your files on the bucket like as follow:

[root@ansible-sdnlab test]# aws s3 ls --recursive nuage-secure-files/5.1.1u1-files
2017-09-22 14:13:12          0 5.1.1u1-files/
2017-09-22 14:13:12    2672108 5.1.1u1-files/libnetwork5.1.1.rpm

Ok, now lets filter that to use it later as follow:

[root@ansible-sdnlab test]# aws s3 ls --recursive nuage-secure-files/5.1.1u1-files | awk '{ print $4 }'
5.1.1u1-files/
5.1.1u1-files/libnetwork5.1.1.rpm

Create your python app to generate URLs

Ok, this is a basic app I’ve done to generate my Signed URL with Expiration date (I’m using 45 days as deadline to expire in seconds):

import sys
import boto
from boto.s3.key import Key
import requests

#setup the bucket
c = boto.connect_s3('AFSGSHDKDKLAHDJDA', 'yeGe/dz9nxxOOxxOOxxxOOxxOOxxO/TI')
b = c.get_bucket('nuage-secure-files', validate=False)

#download the file
k = Key(b)
k.key = sys.argv[1]
#url_prt = k.generate_url( expires_in=259200, force_http=True )
#url_prt = k.generate_url( expires_in=259200, )
url_prt = k.generate_url( expires_in=3888000, )

print url_prt

Ok, now, Let’s create our first Signed URL

[root@ansible-sdnlab test]# python url_keygen.py 5.1.1u1-files/libnetwork5.1.1.rpm
https://nuage-secure-files.s3.amazonaws.com/5.1.1u1-files/libnetwork5.1.1.rpm?Signature=IggFmPRRJlWFvyMHJcYKgYWbXOQ%3D&Expires=1510005892&AWSAccessKeyId=AFAGAFADAFAHGAJKA

And basically you are done.

Playing with ansible

Ok, now, Let’s create a list of Signed URL for all the files (a.k.a keys in AWS’s words)

for i in `aws s3 ls --recursive nuage-secure-files/5.1.1u1-files | awk '{ print $4 }'` ; do python url_keygen.py $i >> signed_url_list.txt; done

After that, It throw that list into a YAML file like this:

rpm_lib_location: "https://nuage-secure-files.s3.amazonaws.com/5.1.1u1-files/libnetwork5.1.1.rpm?Signature=IggFmPRRJlWFvyMHJcYKgYWbXOQ%3D&Expires=1510005892&AWSAccessKeyId={{ AWS_Access_Key }}"

You can add “-e AWS_Access_Key=AFAGAFADAFAHGAJKA” when your invoking your playbook.
And I call the library as follow:

    - name: install library
      yum:
        name: '{{ rpm_lib_location }}'
        state: present
      remote_user: root

That’s all… See ya!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s