Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

scraping using cheerio returns empty array

I am trying to scrape a website https://buff.163.com/market/csgo#tab=buying&page_num=1 using request-promise and cheerio. my entire code is below :

const request = require('request-promise');
const cheerio = require('cheerio');

const url = "https://buff.163.com/market/csgo#tab=buying&page_num=1";

const scrapeArr = [];

async function scrape() {
    try {
        const htmlResult = await request.get(url);
        const $ = await cheerio.load(htmlResult);
        
        $(".card_csgo li")
            .each((i, e) => { 
                const title = $(e).children("h3").text() 
                const link = $(e).children("a").attr("href")
                const scrapeObj = { title , link }
                scrapeArr.push(scrapeObj)
            })
            console.log(scrapeArr);
    } catch(e) {
        console.log(e)
    }
}

scrape()

this results in a empty scrapeArr, The jQuery commands work perfectly in the chrome dev tools console but when I copy paste the same command in cheerio it results in empty array. can someone tell me what’s the problem here?

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

>Solution :

If you open the source of the page, you will see you will see that there are no such DOM elements in HTML. They are generated by browser in runtime, so you need full-fledged browser (and not raw network request + cheerio) to render them.

The good news is that this website has hidden API which you should better use to extract the data you need, just use Chrome Dev Tools to discover it:
https://www.youtube.com/watch?v=kPe3wtA9aPM

Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading