Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Copy file from s3 subfolder in another subfolder in same bucket

I’d like to copy file from subfolder into another subfolder in same s3 bucket. I’ve read lots of questions in SO and I came finally with this code. It has an issue, when I run it it works, but it doesn’t copy the file only, it copy the folder that contain the file into the destination wanted I’ve have the file but inside a folder(root). How do I only copy the files inside that subfolder?

XXXBUCKETNAME:
    -- XXXX-input/  # I want to copy from here
    -- XXXX-archive/ # to here


import boto3
from botocore.config import Config
s3 = boto3.resource('s3', config=Config(proxies={'https': getProperty('Proxy', 'Proxy.Host')}))
bucket_obj = s3.Bucket('XXX')
destbucket = 'XXX'

jsonfiles = []
for obj in bucket_obj.objects.filter(Delimiter='/', Prefix='XXXX-input/', ):
    if obj.key.endswith('json'):
        jsonfiles.append(obj.key)
for k in jsonfiles:
    if k.split("_")[-1:][0] == "xxx.txt":
        dest = s3.Bucket(destbucket)
        source= { 'Bucket' : destbucket, 'Key': k}
        dest.copy(source, "XXXX-archive/"+k)

it give:

XXXBUCKETNAME:
    -- XXXX-input/
    -- XXXX-archive/
        -- XXXX-input/file.txt

I want:

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

XXXBUCKETNAME:
    -- XXXX-input/
    -- XXXX-archive/
        -- file.txt

>Solution :

In S3 there really aren’t any "folders." There are buckets and objects, as explained in documentation. The UI may make it seem like there are folders, but the key for an object is the entire path. So if you want to copy one item, you will need to parse its key and build the destination key differently so that it has the same prefix (path) but end with a different value.

In Amazon S3, buckets and objects are the primary resources, and
objects are stored in buckets. Amazon S3 has a flat structure instead
of a hierarchy like you would see in a file system. However, for the
sake of organizational simplicity, the Amazon S3 console supports the
folder concept as a means of grouping objects. It does this by using a
shared name prefix for objects (that is, objects have names that begin
with a common string). Object names are also referred to as key names.

In your code you are pulling out each object’s key, so that means the key already contains the full "path" even though there isn’t really a path. So you will want to split the key on the / character instead and then take the last element in the resulting list and append that as the file name:

dest.copy(source, "XXXX-archive/" + k.split("/")[-1])
Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading