Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Google Cloud Storage pushing the files into bucket very slowly

I’m using the below code to recursively get all the paths in the folder and push them one by one into the google cloud storage bucket. The problem is, it’s extremely slow. I have around 30-40K files that need to be pushed every day and each one is taking like 0.25 to 0.5 second to push. Is there any way I could push them all together? In bulk? Or another way that makes it way faster?

const {Storage} = require('@google-cloud/storage');
const fs = require('fs');
const path = require('path');

function getAllFilesInDirectoryRecursively(dir){
    const files = fs.readdirSync(dir, {withFileTypes: true});
    for (const file of files) {
        if (file.isDirectory()) {
            yield* getAllFilesInDirectoryRecursively(path.join(dir, file.name));
        } else {
            yield path.join(dir, file.name);
        }
    }
}

const storage = new Storage();

(async function(){
    for (let filePath of getAllFilesInDirectoryRecursively('./main/')) {
        await storage.bucket('bucket.domain.com').upload('./' + filePath, {
            destination: filePath.replace('main', ''),
        });
    }
})()

>Solution :

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

You can use gsutil to upload contents of a directory using:

gsutil cp -r ./main gs://bucket-name

To run this command periodically, you can use a cron job or run the command from NodeJS script after the files have been generated.

For a solution without gsutil, it might be better to use Promise.all() instead of running all upload promises individually.

Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading