I’ve tried to play around with the code to find the links. I’ve narrowed down the HTML to find the relevant images and retreive them all. However, it’s returning an empty list.
I can’t seem to understand where the problem actually.
import requests
from bs4 import BeautifulSoup
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36'}
r = requests.get('https://www.f1-fansite.com/f1-wallpaper/wallpaper-photos-2022-canadian-f1-grand-prix/', headers=headers)
soup = BeautifulSoup(r.content, 'lxml')
pictureslist = soup.find_all('div', attrs={'id': 'gallery-1',
'class': 'gallery galleryid-268780 gallery-columns-3 gallery-size-medium'})
print(pictureslist)
Am I looking at the right place to fetch all image links?
>Solution :
You could select all the gallery-items a and extract the href:
set(e.get('href') for e in soup.select('.gallery-item a'))
Example
import requests
from bs4 import BeautifulSoup
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36'}
r = requests.get('https://www.f1-fansite.com/f1-wallpaper/wallpaper-photos-2022-canadian-f1-grand-prix/', headers=headers)
soup = BeautifulSoup(r.content)
set(e.get('href') for e in soup.select('.gallery-item a'))
Output
{'https://www.f1-fansite.com/wp-content/uploads/2022/06/1019217025-LAT-20220619-GP2209_123003_ONZ8707-725.jpg',
'https://www.f1-fansite.com/wp-content/uploads/2022/06/1019217091-LAT-20220619-GP2209_171345_AF_1474-712.jpg',
'https://www.f1-fansite.com/wp-content/uploads/2022/06/1019217141-LAT-20220619-GP2209_180413_56I3103-923.jpg',
'https://www.f1-fansite.com/wp-content/uploads/2022/06/1019217158-LAT-20220619-GP2209_172751_AF_1595-561.jpg',
'https://www.f1-fansite.com/wp-content/uploads/2022/06/1019217306-SUT-20220619-GP2209_190245DSC_3459-725.jpg',
'https://www.f1-fansite.com/wp-content/uploads/2022/06/17-sunday-canada-2022.jpg',
'https://www.f1-fansite.com/wp-content/uploads/2022/06/18-sunday-canada-2022.jpg',
'https://www.f1-fansite.com/wp-content/uploads/2022/06/19-sunday-canada-2022.jpg',
'https://www.f1-fansite.com/wp-content/uploads/2022/06/20-sunday-canada-2022.jpg',
'https://www.f1-fansite.com/wp-content/uploads/2022/06/2022-Canadian-Grand-Prix-Friday-1.jpg',...}