Unable to scrape data using selenium find_element_by_class_name() in python

November 21, 2021

I am new to webscraping & not a developer nor have any html exp. and was trying to pull some details from my account after logging in into the website but getting errors in find_element_by_class_name()

this is the code I have tried:

from selenium import webdriver

driver = webdriver.Chrome('path/chromedriver.exe')
driver.get("https://www.URL.COM") 

# logged into account manually & maneuvered to the page manually

driver.find_element_by_class_name('css-901oao css-cens5h r-1khnkhu r-13awgt0 r-1oke55r r-1enofrn r-1wzrnnt')

Error

---------------------------------------------------------------------------
NoSuchElementException                    Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_14516/1270707966.py in <module>
----> 1 driver.find_element_by_class_name('css-901oao css-cens5h r-1khnkhu r-13awgt0 r-1oke55r r-1enofrn r-1wzrnnt')

C:\ProgramData\Miniconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py in find_element_by_class_name(self, name)
    562             element = driver.find_element_by_class_name('foo')
    563         """
--> 564         return self.find_element(by=By.CLASS_NAME, value=name)
    565 
    566     def find_elements_by_class_name(self, name):

Also tried

driver.find_element_by_css_selector('css-901oao css-cens5h r-1khnkhu r-13awgt0 r-1oke55r r-1enofrn r-1wzrnnt')

From inspect I was able to view this & tried to extract the one highlighted in the image:

>Solution :

class name expect a single class name. where as you are passing multiple class name here

css-901oao css-cens5h r-1khnkhu r-13awgt0 r-1oke55r r-1enofrn r-1wzrnnt

It won’t work.

Instead

remove the spaces and make a CSS selector out of it.

driver.find_element(By.CSS_SELECTOR, ".css-901oao.css-cens5h.r-1khnkhu.r-13awgt0.r-1oke55r.r-1enofrn.r-1wzrnnt")

Also, Please remember find_element_by_class_name have been deprecated in newest selenium. You should use this instead

find_element(By.CLASS_NAME, "class name")

having said this, the locator that you’ve right now looks brittle in nature. Please use static attribute values.

You could try this xpath

//div[starts-with(@class,'css')]//div[@dir='auto' and contains(@style,'-webkit-line-clamp')]

Please check in the dev tools (Google chrome) if we have unique entry in HTML DOM or not.

xpath that you should check :

//div[starts-with(@class,'css')]//div[@dir='auto' and contains(@style,'-webkit-line-clamp')]

Steps to check:

Press F12 in Chrome -> go to element section -> do a CTRL + F -> then paste the xpath and see, if your desired element is getting highlighted with 1/1 matching node.