I get a website (https://www.diabetesdaily.com/forum/threads/had-a-friend-with-type-one.136015/)
And I found there are many forum pages
Want to use the for loop for web scrapping, therefore could I ask, how I get the maximum number of forum pages on this page by BeaurifulSoup?
Many thanks.
>Solution :
You can try something like this:
import requests
from bs4 import BeautifulSoup as bs
url = "https://www.diabetesdaily.com/forum/threads/had-a-friend-with-type-one.136015/"
req = requests.get(url)
soup = bs(req.content, 'html.parser')
navs = soup.find("ul", { "class" : "pageNav-main" }).find_all("li", recursive=False)
print(navs)
print(f'Length: {len(navs)}')
Result
[<li class="pageNav-page pageNav-page--current"><a href="/forum/threads/had-a-friend-with-type-one.136015/">1</a></li>, <li class="pageNav-page pageNav-page--later"><a href="/forum/threads/had-a-friend-with-type-one.136015/page-2">2</a></li>, <li class="pageNav-page pageNav-page--later"><a href="/forum/threads/had-a-friend-with-type-one.136015/page-3">3</a></li>, <li class="pageNav-page"><a href="/forum/threads/had-a-friend-with-type-one.136015/page-4">4</a></li>]
Length: 4