Sharing #AWS S3 files for ansible thru Signed URLs

Hi Guys, I love automation. However, some much automation should get along with a secure way to share files like qcow2 images and rpm. I am doing that on my own and I would like to share what I’ve learnt
First of all, Thanks to Remi (@rvichery), Guillermo Alvarado (@galvarado89) and Boris Cortes (@boriscortes) for helping me out to find the best method here.

Prepare your system

It’d be great if you create a temporary user/group. My case I created a group called “read_only” like the next picture with the following attached permission “AmazonS3ReadOnlyAccess”

And then I created a user getting the AccessKeyID and the SecretKey. Install aws client. Guess what OS in my case. Nop! it isn’t Mac OSX now, I am doing that in my CentOS 🙂

yum -y install python-pip
pip install aws
pip install boto #checking if it was installed already

Configure your access with: “aws configure” and enter the required info (AccessKeyID and SecretKey)
Now, you can check your files on the bucket like as follow:

[root@ansible-sdnlab test]# aws s3 ls --recursive nuage-secure-files/5.1.1u1-files
2017-09-22 14:13:12          0 5.1.1u1-files/
2017-09-22 14:13:12    2672108 5.1.1u1-files/libnetwork5.1.1.rpm

Ok, now lets filter that to use it later as follow:

[root@ansible-sdnlab test]# aws s3 ls --recursive nuage-secure-files/5.1.1u1-files | awk '{ print $4 }'

Create your python app to generate URLs

Ok, this is a basic app I’ve done to generate my Signed URL with Expiration date (I’m using 45 days as deadline to expire in seconds):

import sys
import boto
from boto.s3.key import Key
import requests

#setup the bucket
c = boto.connect_s3('AFSGSHDKDKLAHDJDA', 'yeGe/dz9nxxOOxxOOxxxOOxxOOxxO/TI')
b = c.get_bucket('nuage-secure-files', validate=False)

#download the file
k = Key(b)
k.key = sys.argv[1]
#url_prt = k.generate_url( expires_in=259200, force_http=True )
#url_prt = k.generate_url( expires_in=259200, )
url_prt = k.generate_url( expires_in=3888000, )

print url_prt

Ok, now, Let’s create our first Signed URL

[root@ansible-sdnlab test]# python 5.1.1u1-files/libnetwork5.1.1.rpm

And basically you are done.

Playing with ansible

Ok, now, Let’s create a list of Signed URL for all the files (a.k.a keys in AWS’s words)

for i in `aws s3 ls --recursive nuage-secure-files/5.1.1u1-files | awk '{ print $4 }'` ; do python $i >> signed_url_list.txt; done

After that, It throw that list into a YAML file like this:

rpm_lib_location: "{{ AWS_Access_Key }}"

You can add “-e AWS_Access_Key=AFAGAFADAFAHGAJKA” when your invoking your playbook.
And I call the library as follow:

    - name: install library
        name: '{{ rpm_lib_location }}'
        state: present
      remote_user: root

That’s all… See ya!

Categories: DevOps

Tagged as: , , , , ,

1 reply »

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: