Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Can't get Table Data from Site After the Page Loads

I’m trying to obtain the 2nd table on the following website. I’ve tried BS4, Pandas, and now selenium, but I can’t obtain the table for the life of me.

The table data doesn’t load until after the page comes up.

There is a dictionary on the ‘view source’ page with the info, but its seems like every element on that page is ‘line-content’, so its makes it difficult to obtain only the desired table info.

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

What is the best way to collect the table data?

from ast import Return
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
import time

path = 'C:\Program Files (x86)\chromedriver.exe'

driver = webdriver.Chrome(path)

driver.get('https://www.linestarapp.com/Ownership/Sport/NFL/Site/DraftKings/PID/295')

time.sleep(5)
driver.maximize_window()

time.sleep(5)

players = driver.find_elements_by_class_name('playerRow').text
print(players)

>Solution :

There are several issues here:

  1. find_elements_by_class_name method returns a list of web elements. You can not apply .text method on a list of web elements, you need to iterate over elements in a list and extract text from each element one by one.
  2. You should not use hardcoded pauses like time.sleep(5) Expected Conditions explicitly waits should be used instead.
    Something like this:
from ast import Return
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time

path = 'C:\Program Files (x86)\chromedriver.exe'

driver = webdriver.Chrome(path)
wait = WebDriverWait(driver, 20)

driver.get('https://www.linestarapp.com/Ownership/Sport/NFL/Site/DraftKings/PID/295')

driver.maximize_window()

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".playerRow")))
#short pause added in order to make sure all the elements are loaded after we know the first element was loaded
time.sleep(0.5)

players = driver.find_elements_by_class_name('playerRow')
for player in players:
    print(player.text)

UPD
Using this css_selector locator #tableTournament .playerRow instead of just .playerRowwill give you the right table rows

Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading