Extracting a var from <script> tag in html

November 30, 2021

I am trying to web scrape product reviews from a page but I’m not sure how to extract a var inside the <script> tags.

Here’s my python code:

import requests
from bs4 import BeautifulSoup
import csv

a_file = open("ProductReviews.csv", "a")
writer = csv.writer(a_file)

# Write the titles of the columns to the CSV file
writer.writerow(["created_at", "reviewer_name", "rating", "content", "source"])

url = 'https://www.lazada.com.my/products/iron-gym-total-upper-body-workout-bar-i467342383.html'

# Connect to the URL
response = requests.get(url)

# Parse HTML and save to BeautifulSoup object
soup = BeautifulSoup(response.content, "html.parser")

data = soup.findAll('script')[123]

if 'var __moduleData__' in data.string:
    print("Yes")

Here’s the page source (I removed the unnecessary code):

<html>
<head>
    <title></title>
</head>
<body>

    <script>
        var __moduleData__ = {
        "data": {
            "root": {
                "fields": {
                    "review": {
                        "reviews": [{
                            "rating": 5,
                            "reviewContent": "tq barang dah sampai",
                            "reviewTime": "24 May 2021",
                            "reviewer": "Jaharinbaharin",

                        }, {
                            "rating": 5,
                            "reviewContent": "Beautiful quality👌👌👌",
                            "reviewTime": "08 Sep 2021",
                            "reviewer": "M***.",

                        }, {
                            "rating": 5,
                            "reviewContent": "the box was badly dented but the item was intact...just that my door frame is shallow and slippery....I can't pull up without worrying of falling down",
                            "reviewTime": "25 Aug 2021",
                            "reviewer": "David S.",

                        }, {
                            "rating": 5,
                            "reviewContent": "Haven’t really opened it yet but please put some effort on the packaging for future improvement thanks it was really fast",
                            "reviewTime": "14 Dec 2020",
                            "reviewer": "Yasir A.",
                        
                        }, {
                            "rating": 5,
                            "reviewContent": "Seems to be ok, good quality.. No weight restriction mentioned on the box.. I'm about 90kg, it could handle my weight so far..",
                            "reviewTime": "22 May 2020",
                            "reviewer": "Kevin",
                        }]
                    },
                }
            },
        },
    };
  </script>

</body>
</html>

I just want to get the review data only so I’d like to know how to extract the value of var __moduleData__.

>Solution :

You can use a regex to select your variable:

json.loads(re.search(r'var __moduleData__ = ({.*})', response.text).group(1))

Example

from bs4 import BeautifulSoup
import json,re,requests

url = 'https://www.lazada.com.my/products/iron-gym-total-upper-body-workout-bar-i467342383.html'
response = requests.get(url)

d = json.loads(re.search(r'var __moduleData__ = ({.*})', response.text).group(1))

d['data']['root']['fields']['seller']

Output

{'chatResponsiveRate': {'labelText': 'Chat Response', 'value': '100%'},
 'chatUrl': 'https://pages.lazada.com.my/wow/i/my/im/chat?brandId=21411',
 'hideAllMetrics': False,
 'imEnable': True,
 'imUserId': '100285367',
 'name': 'MR SIX PACK',
 'newSeller': False,
 'percentRate': '96%',
 'positiveSellerRating': {'labelText': 'Seller Ratings', 'value': '96%'},
 'rate': 0.96,
 'rateLevel': 3,
 'sellerId': '1000052649',
 'shipOnTime': {'labelText': 'Ship On Time', 'value': '97%'},
 'shopId': 255007,
 'size': 5,
 'time': 2,
 'type': '4',
 'unit': 'years',
 'url': '//www.lazada.com.my/shop/mr-six-pack/?itemId=467342383&channelSource=pdp'}