Python Request Takes Forever – Can't find proper XHR Request

September 14, 2022

I’m want to access some data from:

https://www.calorie-charts.info/food/all/banana

I tried a Python Request Session but it takes up to 2 minutes to get a response.

import requests

s = requests.Session()

headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}

url = 'https://www.calorie-charts.info/food/all/banana"'

s.headers.update(headers)

r = s.get(url)
print(r.text)

I also tried to find an api link for a XHR Request but I couldn’t find one in the Dev Tool – Network Tab.

How can I speed up the process or find a link for a XHR Request?

>Solution :

This is one way to get that data (UPDATED: all 34 pages):

import requests
from bs4 import BeautifulSoup as bs
import pandas as pd
from tqdm import tqdm

headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}

s = requests.Session()
s.headers.update(headers)

big_df = pd.DataFrame()

for x in tqdm(range(1,35)):
    url = f'https://www.calorie-charts.info/food/all/banana/page/{x}'

    r = s.get(url)
    df = pd.read_html(r.text)[0]
    big_df = pd.concat([big_df, df], axis=0, ignore_index=True)
print(big_df)

Result:

    Name    Calories(kcal)  Amount  Energy (kj) Proteins (g)    Carbohydrates (g)   Fat (g) Fiber (g)   Unnamed: 8
0   banana  84.5    medium banana (90 g)    354 1   20.0    22.0    2.0 Detail
1   banana bio  105.0   126 g   440 1   27.0    38.0    3.0 Detail
2   banana bar  147.6   40 g    618 6   28.0    4.0 NaN Detail
3   banana curd 224.7   100 g   941 18  30.0    3.0 1.0 Detail
4   ripe banana 199.6   200 g   836 2   40.0    1.0 4.0 Detail
... ... ... ... ... ... ... ... ... ...
1675    Oatmeal lumps of cheese 3 bananas, 1 cheese, 2...   20.5    piece (10 g)    86  78.0    3.0 73.0    36.0    Detail
1676    pancakes (tangerine, apple, banana, 1/2 curd, ...   54.9    1 pancake (71 g)    230 5.0 6.0 1.0 99.0    Detail
1677    Fit cake with nuts, avocados and bananas, almo...   264.4   100 g   1 107   7.0 30.0    13.0    6.0 Detail
1678    100% Whey Protein vanilla, banana, strawberry,...   113.4   portion (30 g)  475 23.0    2.0 1.0 42.0    Detail
1679    Excellent 24% Protein Bar pineapple with cocon...   372.8   85 g    1 561   20.0    36.0    16.0    2.0 Detail

Requests documentation: https://requests.readthedocs.io/en/latest/

Also pandas relevant documentation: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_html.html