I aimed to scrape address data from the website that indicated in the code below and i encountered with this error.
File "c:\Users\efede\OneDrive\Masaüstü\Scrapping\pythonfiles\gCharge.py", line 15, in <module>
WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,'#map_canvas > div > div > div:nth-child(2) > div:nth-child(2) > div > div:nth-child(4) > div > div > div > div.gm-style-iw.gm-style-iw-c > div > div > table > tbody > tr:nth-child(1) > td:nth-child(2) > h5')))
File "C:\Users\efede\OneDrive\Masaüstü\Scrapping\scrappingenv\Lib\site-packages\selenium\webdriver\support\wait.py", line 95, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
CODE:
from bs4 import BeautifulSoup
import pandas as pd
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
path = r"C:\Users\efede\OneDrive\Masaüstü\exeler\chromedriver.exe"
options = webdriver.ChromeOptions()
options.add_experimental_option('excludeSwitches', ['enable-logging'])
driver = webdriver.Chrome(options=options)
driver.get("https://www.g-charge.com.tr/Map.Aspx")
WebDriverWait(driver,30).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,'#map_canvas > div > div > div:nth-child(2) > div:nth-child(2) > div > div:nth-child(4) > div > div > div > div.gm-style-iw.gm-style-iw-c > div > div > table > tbody > tr:nth-child(1) > td:nth-child(2) > h5')))
kaynak=driver.page_source
soup=BeautifulSoup(kaynak,"html.parser")
basliklar= driver.find_elements(By.CSS_SELECTOR,'#map_canvas > div > div > div:nth-child(2) > div:nth-child(2) > div > div:nth-child(4) > div > div > div > div.gm-style-iw.gm-style-iw-c > div > div > table > tbody > tr:nth-child(1) > td:nth-child(2) > h5')
titles= [baslik.text for baslik in basliklar]
print(titles)
I tried to look at the other questions in this website but haven’t foun anything useful yet. How can i fix this?
>Solution :
This seems awfully close to an XY Problem. While not explicitly stated, you are after the charger details from that map. Selenium is not the appropriate tool of choice here. You can obtain that information by inspecting the Network calls made by JS on that page, and scraping the API endpoint where the info is sourced from:
import requests
import pandas as pd
headers= {
'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36'
}
r = requests.get("https://www.g-charge.com.tr/Services/Charger.aspx?Operation=GetChargers", headers=headers)
df = pd.json_normalize(r.json())
print(df)
Result in terminal:
Id ChargerNumber ChargerTitle ChargerAddress ChargerStatus ChargerType ChargerLat ChargerLng IsActive PinType
0 39 Mod-3 Tip-2 22kW & Mod-1/2 3,7kW RFID İSPARK Haldun Alagaş İSPARK Haldun Alagaş OFF Mod-3 Tip-2 & Mod-1 Schuko 41.02620760142811 29.105417132377625 True dev
1 48 GCS11623BNA1WN Brisa Akademi Brisa Akademi OFF Mod-3 Tip-2 40.75996813647878 29.98842716217041 True dev
2 53 2 x Mod-3 Tip-2 3,7kW KKB KKB OFF Mod-3 Tip-2 40.99726137817314 29.099231958389282 True dev
3 55 2 x Mod-3 Tip-2 22kW RFID Ofishane Ofishane OFF Mod-3 Tip-2 41.084529 28.979847000000063 True dev
4 57 2 x Mod-3 Tip-2 22kW RFID Lounge-2 Lounge-2 OFF Mod-3 Tip-2 41.04003463723342 28.764374256134033 True dev
... ... ... ... ... ... ... ... ... ... ...
111 6218 Mod-4 Tip-2 CCS 180(2x90)kW TAMTEX TEKSTİL TAMTEX TEKSTİL OFF Mod-4 Tip-2 CCS 41.209098 27.844295 True dev
112 6219 Mod-4 Tip-2 CCS 120(2x60)kW GERSAN ELEKTRİK ÇAYCUMA GERSAN ELEKTRİK ÇAYCUMA OFF Mod-4 Tip-2 CCS 41.4073 32.127965 True dev
113 6220 Mod-3 Tip-2 2x22kW Safir Garage Otomotiv Turizm Tic. Ltd. Şti. Safir Garage Otomotiv Turizm Tic. Ltd. Şti. OFF Mod-3 Tip-2 40.929214 29.308886 True dev
114 6221 Mod-3 Tip-2 22kW CİHAN ET KASAP BORNOVA CİHAN ET KASAP BORNOVA OFF Mod-3 Tip-2 38.448616 27.208421 True dev
115 6222 Mod-3 Tip-2 22kW TUZ TERAPİ MERKEZİ-Tuzluca Kaymakamlığı TUZ TERAPİ MERKEZİ-Tuzluca Kaymakamlığı OFF Mod-3 Tip-2 40.049999 43.66459 True dev
116 rows × 10 columns