I’m trying to scrape a specific class of divs from an HTML document using bs4. However, when I use the find_all() method, I do not get back the divs that I want despite the fact that I can see those divs when I print out the text of the soup object I have.
I’m looking for a div like this from this website
<div style="top:128px;left:58px;" tip="Nov 20, 2009, OKC vs WAS<br>4th Qtr, 11:02 remaining<br>Made 2-pointer from 20 ft<br>OKC now leads 97-86" class="tooltip make">●</div>
to do this I search the soup object
shots = soup.find_all("div", _class="tooltip make")
However, the returning shots array is empty
The divs are loaded through JS on the website, however even when I used selenium to render the page this same problem occurred. Additionally I’ve printed out the soup document I recive when just making the HTTP request and I can see the divs I’m looking for when I print it out.
This one has had me stuck for a bit so I would appreciate any help!
>Solution :
The section you are looking for is commented out, so easiest way would be uncomment the elements by replacement of <!-- and --> from the response text:
requests.get('https://www.basketball-reference.com/players/h/hardeja01/shooting/2010').text.replace('<!--','').replace('-->','')
Also check your selection and change the class parameter:
find_all("div", class_="tooltip make")
or use css selectors:
select('div.tooltip.make)
Example
import requests
from bs4 import BeautifulSoup
soup = BeautifulSoup(requests.get('https://www.basketball-reference.com/players/h/hardeja01/shooting/2010').text.replace('<!--','').replace('-->','')).find_all("div", class_="tooltip make")
soup