Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Problems scraping dynamic content with requests and BeautifulSoup

I have tried to scrape the response of the form on the website https://www.languagesandnumbers.com/how-to-count-in-german/en/deu/ by trying to fill out the form and submitting it with requests and BeautifulSoup. After inspecting network-traffic of the submit, I found out that the post params are "numberz" and "lang". That’s why I tried to post the following:

import requests
from bs4 import BeautifulSoup

with requests.Session() as session:
    response = session.post('https://www.languagesandnumbers.com/how-to-count-in-german/en/deu/', data={
        "numberz": "23",
        "lang": "deu"
    })

    soup = BeautifulSoup(response.content, "lxml")
    print(soup.find(id='words').get_text())

Unfortunately, the response is dynamic and not visible, so after submitting the form I always get the main page back without any text in the particular div which actually carries that response. Is there another way to scrape the response using requests and BeautifulSoup and not use selenium?

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

>Solution :

You do not need BeautifulSoup but the correct url to get only the result of written number:

https://www.languagesandnumbers.com/ajax/en/

Cause it returns in this way ack:::dreiundzwanzig you hav to extract the string:

response.text.split(':')[-1]

Example

import requests

with requests.Session() as session:
    response = session.post('https://www.languagesandnumbers.com/ajax/en/', data={
        "numberz": "23",
        "lang": "deu"
    })
response.text.split(':')[-1]

Output

dreiundzwanzig
Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading