how to pass contents of a file line by line to requests module in python

April 18, 2022

I have written this script that will retrieve the contents of a web page.

import requests
import bs4

with requests.session() as r:
    r = requests.get("https://www.example.com")
    response = r.text
    print(response)

However, I have a list of URLs in a text file. Is there any way I can pass the contents of this file directly to requests.get() instead of typing each one manually.

>Solution :

Just put it all in a loop.

import requests
import bs4

text_file_name = "list_of_urls.txt"

with requests.session() as session:
    with open(text_file_name) as file:
        for line in file:
            url = line.strip()
            if url:
                resp = session.get(url)
                response = resp.text
                print(response)

note: you weren’t using the requests session object, so fixed that.