Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Webscraping: Loop through multiple urls

I have successfully scrapped several websites individually.
However, now I want to have a single script so that I don’t have to run each script individually all the time.
I would like to build a for loop that goes through all websites and replaces the x with a string.
Unfortunately, there are no numbers, with which I could go through the individual pages with "for x in range", but there are just the strings mentioned.

Here is my current code:

from bs4 import BeautifulSoup
import requests
import pandas as pd
    

movielist = []

for x in ... ('action', 'comedy', 'thriller', 'drama', 'sport'): # what should i insert instead of ...?
    r = requests.get(f'https://movie.com/{x}', headers=headers)
    soup = BeautifulSoup(r.text, 'html.parser')
    spiele = soup.find_all('div', {'class': 'row'})

The site is not real, its just a question how to do that.

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

I am very happy about your help, thank you very much.

>Solution :

Just remove the …
Your tuple is iterable to you can go through every element like this:

for x in ('action', 'comedy', 'thriller', 'drama', 'sport'): # what should i insert instead of ...?
    print(f'https://movie.com/{x}')

output:

https://movie.com/action
https://movie.com/comedy
https://movie.com/thriller
https://movie.com/drama
https://movie.com/sport
Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading