How to get the http Header Information for web scraping

May 20, 2023

I am new to web scraping,
How can i get the product ID from the HTTP header (screenshot attached)

source : https://www.pickaboo.com/product-detail/samsung-galaxy-a03-3gb-32gb/

I am using requests to get the information but still no luck.

url = 'https://www.pickaboo.com/product-detail/samsung-galaxy-a04-3gb-32gb/'
requests.get(url)

I got all the information except review section.

Screenshot

>Solution :

The product ID is stored on the main page inside <script> element. To get it you can use next example:

import json
import requests
from bs4 import BeautifulSoup

url = 'https://www.pickaboo.com/product-detail/samsung-galaxy-a03-3gb-32gb/'

soup = BeautifulSoup(requests.get(url).content, 'html.parser')
data = soup.select_one('#__NEXT_DATA__')
data = json.loads(data.text)

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

print(data['props']['pageProps']['product']['id'])

Prints:

Note: On your screenshot is not shown HTTP header but URL parameter.