Writing big-json stream to a file – why is it so slow?

October 4, 2022

I have a simple script that creates a large object and tries to save it in a file as a json. When I use big-json library it takes x4 more time than if I used just JSON.stringify – why is that ?

    import json from 'big-json'

    const myData = [];
    for (var i = 0; i < 10000000; ++i) {
        myData.push({'id': i, 'name': 'SomeName'})
    }
    const fetchAt = new Date()
    const myDataFileWriteStream = fs.createWriteStream('./myfile.json')
    console.log("Starting saving ...")
    json.createStringifyStream({body: myData})
        .pipe(myDataFileWriteStream)
        .on('finish', function () {
            const writeAt = new Date();
            console.log(`Data written in ${(writeAt - fetchAt) / 1000} seconds.`);
        });

// this takes like x4 time than if I did just
//  fs.writeFileSync('./myfile.json', JSON.stringify(myData));

How can I improve its performance ?

>Solution :

When I use big-json library it takes x4 more time than if I used just JSON.stringify – why is that?

Because it’s a tradeoff you make by choosing big-json.

The JSON.stringify() API stringifies the (big) JSON object into memory, and you write it to a file from memory afterwards.

big-json writes it to the stream as it goes, without needing to spend extra memory for the in-memory JSON representation. Also, big-json is written in JavaScript (well, TypeScript); the Node.js intrinsic JSON things are written in C(++), which is much faster than JavaScript.