I am trying to scrape all match reports links from the page but there is ‘load more’ button, and I don’t want to use selenium. Is there any solution to collect all links without selenium.
Thanks in advance.
Here what I tried:
from bs4 import BeautifulSoup as bs
import requests
r=requests.get('https://www.iplt20.com/news/match-reports')
soup = bs(r.text,'lxml')
for match in soup.find_all('div',class_='latest-slider-wrap
position-relative'):
links = match.find('a')
print(links['href'])
>Solution :
Try:
import requests
from bs4 import BeautifulSoup
url = "https://www.iplt20.com/news/match-reports"
soup = BeautifulSoup(requests.get(url).content, "html.parser")
for a in soup.select("#div-match-report a:has(li)"):
print(a["href"])
Prints:
https://www.iplt20.com/news/4014/tata-ipl-2024-match-11-lsg-vs-pbks-match-report
https://www.iplt20.com/news/4012/tata-ipl-2024-match-10-rcb-vs-kkr-match-report
https://www.iplt20.com/news/4011/tata-ipl-2024-match-09-rr-vs-dc-match-report
https://www.iplt20.com/news/4009/tata-ipl-2024-match-08-srh-vs-mi-match-report
https://www.iplt20.com/news/4007/tata-ipl-2024-match-07-csk-vs-gt-match-report
https://www.iplt20.com/news/4006/tata-ipl-2024-match-06-rcb-vs-pbks-match-report
https://www.iplt20.com/news/4004/tata-ipl-2024-match-05-gt-vs-mi-match-report
https://www.iplt20.com/news/4003/tata-ipl-2024-match-04-rr-vs-lsg-match-report
https://www.iplt20.com/news/4001/tata-ipl-2024-match-03-kkr-vs-srh-match-report
https://www.iplt20.com/news/4000/tata-ipl-2024-match-02-pbks-vs-dc-match-report
https://www.iplt20.com/news/3999/tata-ipl-2024-match-01-csk-vs-rcb-match-report
https://www.iplt20.com/news/3976/tata-ipl-2023-final-csk-vs-gt-match-report
https://www.iplt20.com/news/3974/tata-ipl-2023-qualifier-2-gt-vs-mi-match-report
https://www.iplt20.com/news/3973/tata-ipl-2023-eliminator-lsg-vs-mi-match-report
https://www.iplt20.com/news/3972/tata-ipl-2023-qualifier-1-gt-vs-csk-match-report
https://www.iplt20.com/news/3971/tata-ipl-2023-match-70-rcb-vs-gt-match-report
https://www.iplt20.com/news/3970/tata-ipl-2023-match-69-mi-vs-srh-match-report
https://www.iplt20.com/news/3969/tata-ipl-2023-match-68-kkr-vs-lsg-match-report
https://www.iplt20.com/news/3968/tata-ipl-2023-match-67-dc-vs-csk-match-report
https://www.iplt20.com/news/3967/tata-ipl-2023-match-66-pbks-vs-rr-match-report
https://www.iplt20.com/news/3966/tata-ipl-2023-match-65-srh-vs-rcb-match-report
EDIT: To get all links you can use their Ajax pagination API:
import requests
api_url = "https://www.iplt20.com/add-more-match-report?page={page}&type=match-reports"
for page in range(1, 4): # <-- adjust number of pages here
print(f"{page=}")
data = requests.get(api_url.format(page=page)).json()
for d in data["newsResponce"]["data"]:
print(f'https://www.iplt20.com/news/{d["id"]}/{d["titleUrlSegment"]}')
Prints:
...
page=2
https://www.iplt20.com/news/3964/tata-ipl-2023-match-64-pbks-vs-dc-match-report
https://www.iplt20.com/news/3963/tata-ipl-2023-match-63-lsg-vs-mi-match-report
https://www.iplt20.com/news/3962/tata-ipl-2023-match-62-gt-vs-srh-match-report
https://www.iplt20.com/news/3960/tata-ipl-2023-match-61-csk-vs-kkr-match-report
https://www.iplt20.com/news/3959/tata-ipl-2023-match-60-rr-vs-rcb-match-report
https://www.iplt20.com/news/3958/tata-ipl-2023-match-59-dc-vs-pbks-match-report
https://www.iplt20.com/news/3956/tata-ipl-2023-match-58-srh-vs-lsg-match-report
https://www.iplt20.com/news/3955/tata-ipl-2023-match-57-mi-vs-gt-match-report
https://www.iplt20.com/news/3953/tata-ipl-2023-match-56-kkr-vs-rr-match-report
https://www.iplt20.com/news/3952/tata-ipl-2023-match-55-csk-vs-dc-match-report
https://www.iplt20.com/news/3951/tata-ipl-2023-match-54-mi-vs-rcb-match-report
https://www.iplt20.com/news/3947/tata-ipl-2023-match-53-kkr-vs-pbks-match-report
https://www.iplt20.com/news/3946/tata-ipl-2023-match-52-rr-vs-srh-match-report
https://www.iplt20.com/news/3945/tata-ipl-2023-match-51-gt-vs-lsg-match-report
https://www.iplt20.com/news/3944/tata-ipl-2023-match-50-dc-vs-rcb-match-report
https://www.iplt20.com/news/3943/tata-ipl-2023-match-49-csk-vs-mi-match-report
https://www.iplt20.com/news/3942/tata-ipl-2023-match-48-rr-vs-gt-match-report
https://www.iplt20.com/news/3940/tata-ipl-2023-match-47-srh-vs-kkr-match-report
https://www.iplt20.com/news/3938/tata-ipl-2023-match-46-pbks-vs-mi-match-report
https://www.iplt20.com/news/3937/tata-ipl-2023-match-45-lsg-vs-csk-match-report
https://www.iplt20.com/news/3936/tata-ipl-2023-match-44-gt-vs-dc-match-report
page=3
https://www.iplt20.com/news/3934/tata-ipl-2023-match-43-lsg-vs-rcb-match-report
https://www.iplt20.com/news/3932/tata-ipl-2023-match-42-mi-vs-rr-match-report
https://www.iplt20.com/news/3931/tata-ipl-2023-match-41-csk-vs-pbks-match-report
https://www.iplt20.com/news/3930/tata-ipl-2023-match-40-dc-vs-srh-match-report
...