Selenium and bs4 can't retrieve element on web page

October 2, 2022

I need to retrieve the country name in each rider’s page. Sometimes this code works and sometimes it doesn’t (soup.find() return None). Why?

from selenium import webdriver
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import time

names = ['Fabio+Di Giannantonio', 'Francesco+Bagnaia']
for name in names:
    driver = webdriver.Chrome("/usr/bin/chromedriver")

    driver.get(f"https://www.motogp.com/en/riders/profile/{name}")
    soup = BeautifulSoup(driver.page_source)
    print(soup.find("p", "card-text c-rider-country").get_text())
    time.sleep(30)
    driver.close()

>Solution :

I’m not familiar with BeautifulSoup so I’ll give a Selenium solution.
With Selenium your code is missing a wait – you need to wait for element to be fully rendered before extracting it’s text.
The best practice to do that with Selenium is to use WebDriverWait, as following:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("start-maximized")

webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(service=webdriver_service, options=options)
wait = WebDriverWait(driver, 10)

names = ['Fabio+Di Giannantonio', 'Francesco+Bagnaia']
for name in names:
    driver.get(f"https://www.motogp.com/en/riders/profile/{name}")
    title =wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "p.card-text.c-rider-country"))).text
    print(title)

The output is stable: