Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Response Buffer data is truncated in Node JS

I’m trying to figure out how to use the request function of the http and https built-in modules in Node JS (I am aware of third-party wrappers but I’m trying to figure it out myself). I’m running into an issue where the Buffer data from the Response is partially cut off at the end. The issue does not occur when testing with cURL.

Here is the code I’ve been using:

const { request: httpRequest } = require("http");
const { request: httpsRequest } = require("https");
const parseURLData = (url, init) => {
  const { hostname, protocol, port: somePort, pathname, search } = new URL(url);
  const port = +somePort || (protocol === "https:" ? 443 : 80);
  const options = { hostname, port, path: pathname + search, ...init };
  return [protocol, options];
};
const makeRequest = (url, init = { method: "GET" }) => {
  const [protocol, options] = argumentsToOptions(url, init);
  const request = protocol === "http:" ? httpRequest : httpsRequest;
  return new Promise((resolve, reject) =>
    request(options, (res) => {
      res.on("error", reject);
      resolve(res);
    })
      .on("error", reject)
      .end()
  );
};
// not using `async/await` while testing
makeRequest("https://jsonplaceholder.typicode.com/users/1/")
  .then((res) => 
    new Promise((resolve) =>
      res.on("data", (buffer) => {
        resolve(buffer.toString("utf8")); // part of data is cut off
        // resolve(JSON.parse(buffer.toString()));
      })
    )
  )
  .then(console.log)
  .catch(console.error);

Here is the expected output (from cURL):

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

{
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret",
  "email": "Sincere@april.biz",
  "address": {
    "street": "Kulas Light",
    "suite": "Apt. 556",
    "city": "Gwenborough",
    "zipcode": "92998-3874",
    "geo": {
      "lat": "-37.3159",
      "lng": "81.1496"
    }
  },
  "phone": "1-770-736-8031 x56442",
  "website": "hildegard.org",
  "company": {
    "name": "Romaguera-Crona",
    "catchPhrase": "Multi-layered client-server neural-net",
    "bs": "harness real-time e-markets"
  }
}

And here is the actual output, which for some reason is slightly different each time I run the code:

{
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret",
  "email": "Sincere@april.biz",
  "address": {
    "street": "Kulas Light",
    "suite": "Apt. 556",
    "city": "Gwenborough",
    "zipcode": "92998-3874",
    "geo": {
      "lat": "-37.3159",
      "lng": "81.1496"
    }
  },
  "p

What is the correct solution to this problem? How can I get the all of the data from the request?

>Solution :

The buffer should only be processed when the end event fires, otherwise you may be processing an incomplete buffer.

makeRequest("https://jsonplaceholder.typicode.com/users/1/")
  .then((res) => 
    new Promise((resolve) => {
        let totalBuffer = "";

        res.on("data", (buffer) => {
            totalBuffer += buffer.toString("utf8");    
        });

        res.on("end", () => resolve(totalBuffer));
    })
  )
  .then(console.log)
  .catch(console.error);

The response is almost always truncated into several pieces when the file exceeds 1mb, so it is necessary to use the end event which indicates that all available data has been handled by the stream.

https://nodejs.org/api/http.html
Look for "JSON fetching example"

Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading