Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

How to scrape data from multiple pages using beautifulsoup

I am try to scrape multiple page but they give me nothing kindly help me to resolve these issue

    import requests
    from bs4 import BeautifulSoup
    import pandas as pd
    headers= {'User-Agent': 'Mozilla/5.0'}
    for page in range(1,2 ):
        response = requests.get("https://www.avbuyer.com/aircraft/private-jets={page}".format(
                page=page
            ),
            headers=headers,
        )
    
        soup = BeautifulSoup(response.content, 'html.parser')
        postings = soup.find_all('div', class_ = 'listing-item premium')
        for post in postings:
            link = post.find('a', class_ = 'more-info').get('href')
            link_full = 'https://www.avbuyer.com'+ link
            plane = post.find('h2', class_ = 'item-title').text
            price = post.find('div', class_ = 'price').text
            location = post.find('div', class_ = 'list-item-location').text
            print(location)

>Solution :

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

problem was in page = that should be page- .Now your code is working fine.

import requests
from bs4 import BeautifulSoup
import pandas as pd
headers = {'User-Agent': 'Mozilla/5.0'}
for page in range(1, 2):
    response = requests.get("https://www.avbuyer.com/aircraft/private-jets/page-{page}".format(page=page),headers=headers,)

    soup = BeautifulSoup(response.content, 'html.parser')
    postings = soup.find_all('div', class_='listing-item premium')
    for post in postings:
        link = post.find('a', class_='more-info').get('href')
        link_full = 'https://www.avbuyer.com' + link
        plane = post.find('h2', class_='item-title').text
        price = post.find('div', class_='price').text
        location = post.find('div', class_='list-item-location').text
        print(location)

Output:

North America + Canada, United States - MD, For Sale by Avpro Inc.
North America + Canada, United States - WI, For Sale by Lone Mountain Aircraft Sales   
North America + Canada, United States - MD, For Sale by Avpro Inc.
North America + Canada, United States - MD, For Sale by Avpro Inc.
Europe, Monaco, For Sale by Global Jet Monaco
South America, Puerto Rico, For Sale by JetHQ
North America + Canada, United States - NE, For Sale by Duncan Aviation
North America + Canada, United States - DE, For Sale by Leading Edge Aviation Solutions
North America + Canada, United States - TX, For Sale by Par Avion Ltd.
North America + Canada, United States - MD, For Sale by Avpro Inc.
Europe, Switzerland, For Sale by Jetcraft
Europe, United Kingdom - England, For Sale by Jets4UDirect Ltd
North America + Canada, United States - MD, For Sale by Avpro Inc.
North America + Canada, United States - MT, For Sale by SkyWorld Aviation
North America + Canada, United States - MD, For Sale by Avpro Inc.
North America + Canada, United States - AZ, For Sale by Hatt & Associates
Europe, Switzerland, For Sale by Jetcraft
North America + Canada, United States - MD, For Sale by Avpro Inc.
North America + Canada, United States - MD, For Sale by Avpro Inc.
North America + Canada, United States - MD, For Sale by Avpro Inc.
Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading