So I’m using Puppeteer to scrap text from social media, I want to only scrap the text from a post, when I use the chrome developer tool to read what is the class name of the div which contains the text, it always displays a different class name when I reload the page but stay on the same post(see image)

But I noticed that the div class name always ends with .text-content, is there a way to select a div with only the end of the class name?

I tried to use the $ selector like this :

document.querySelectorAll("[class$='text-content']")

And yes it finds the correct div but if I try to use .textContent or .innerText it doesn’t work and it returns undefined.

I also tried to select all divs from the developer console and then see if I could use the index of this div but it turns out that the index also changes every time I reload the page

What I wrote in the developer console :

document.querySelectorAll('div')

and then it gave me an array of divs but as I said I can’t use that if the index changes every time.

>Solution :

Why your solution won’t work

Document.querySelectorAll will return an HTMLCollection (an array-like element) so accessing to Node.textContent property will result in undefined, you should either use Document.querySelector or get the first index separately.

Get individual element

Working example for demonstration:

document.querySelectorAll("[class$='text-content']")[0].textContent

const content = document.querySelector("[class$='text-content']").textContent;

console.log(content)

<div class="MuiBox-root jss1500 jss1169 js1499 text-content">This is the content</div>

or

document.querySelector("[class$='text-content']").textContent

const content = document.querySelectorAll("[class$='text-content']")[0].textContent;

console.log(content)

<div class="MuiBox-root jss1500 jss1169 js1499 text-content">This is the content</div>

Get all the matching elements

Also if you want to get all of the you can do a loop over the elements provided by querySelectorAll and the with the help of Array#forEach.

const elements = document.querySelectorAll("[class$='text-content']");

Array.from(elements).forEach(element => console.log(element.textContent))

<div class="MuiBox-root jss1500 jss1169 js1499 text-content">This is the content</div>
<div class="MuiBox-root jss1500 jss1169 js1499 text-content">This is the content 2</div>
<div class="MuiBox-root jss1500 jss1169 js1499 text-content">This is the content 3</div>
<div class="MuiBox-root jss1500 jss1169 js1499 text-content">This is the content 4</div>
<div class="MuiBox-root jss1500 jss1169 js1499 text-content">This is the content 5</div>
<div class="MuiBox-root jss1500 jss1169 js1499 text-content">This is the content 6</div>