Selenium cant puul the products prices from a website

February 20, 2022

I have a program that reads data from a website using selenium. In this case I want to get the prices of the perfumes.

for i in driver.find_elements_by_css_selector("spam.items_show_price_text"):
    print(i.text)

When I run the program it prints for me 139 empty lines(139 is the number of the perfumes that are in the website).
I guess it has to do with the "span.items_show_price_text" parameter of the driver.find_elements_by_css_selector function.

The whole code (there is no need to understand the while loop, it is basicly runs the program for all of the page, because it is loding when you scroll down)

from datetime import date

import gspread
from oauth2client.service_account import ServiceAccountCredentials
from selenium import webdriver
from selenium.webdriver import ChromeOptions
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

urlM = 'https://www.myperfume.co.il/155567-%D7%9B%D7%9C-%D7%94%D7%9E%D7%95%D7%AA%D7%92%D7%99%D7%9D-%D7%9C%D7%92%D7%91' \
       '%D7%A8?order=up_title&page=0'
urlF = 'https://www.myperfume.co.il/155569-%D7%9B%D7%9C-%D7%94%D7%9E%D7%95%D7%AA%D7%92%D7%99%D7%9D-%D7%9C%D7%90%D7%99' \
       '%D7%A9%D7%94?order=up_title&page=0'

scope = ["https://spreadsheets.google.com/feeds", 'https://www.googleapis.com/auth/spreadsheets',
         "https://www.googleapis.com/auth/drive.file", "https://www.googleapis.com/auth/drive"]
creds = ServiceAccountCredentials.from_json_keyfile_name("credentials.json", scope)
client = gspread.authorize(creds)

spreadsheet = client.open("Perfumes")

options = ChromeOptions()
options.headless = True
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)

namesM = []
pricesM = []
namesMe = ' '

lenM = 0

num = 0
while len(namesMe) != 0:
    urlM = urlM[:-1] + str(int(urlM[-1]) + 1)

    # ---=MALE=---
    driver.get(urlM)
    # names
    for i in driver.find_elements_by_css_selector("h3.title.text-center"):
        lenM += 1
        namesM.append(i.text)
    # prices
    for i in driver.find_elements_by_css_selector("spam.items_show_price_text"):
    print(i.text)
        print(i.text)

>Solution :

As I see on that site, the locator for actual prices (after the price off) will be

'span.price'

So, instead of

for i in driver.find_elements_by_css_selector("spam.items_show_price_text"):

Try using

for i in driver.find_elements_by_css_selector("span.price"):

You should also add waits/ delays there.
Again, it is preferably to use Expected Conditions explicit waits for that.
And don’t mess spam with span 🙂