How to scrap all webpage successfully?

May 3, 2023

I tried to scrap a webpage and I get unrelevant texts instead of html tags whereas request’s status code is 200.

from bs4 import BeautifulSoup
import requests

url = "https://understat.com/league/La_liga"

res = requests.get(url)
content = BeautifulSoup(res.text,"html.parser")

print(content.prettify())

Unrelevant texts:

x22,\x22games\x22\x3A\x2226\x22,\x22time\x22\x3A\x222086\x22,\x22goals\x22\x3A\x220\x22,\x22xG\x22\x3A\x220.7800159454345703\x22,\x22assists\x22\x3A\x220\x22,\x22xA\x22\x3A\x220.21609876118600368\x22,\x22shots\x22\x3A\x229\x22,\x22key_passes\x22\x3A\x224\x22,\x22yellow_cards\x22\x3A\x222\x22,\x22red_cards\x22\x3A\x222\x22,\x22position\x22\x3A\x22D\x20S\x22,\x22team_title\x22\x3A\x22Real\x20Betis\x22,\x22npg\x22\x3A\x220\x22,\x22npxG\x22\x3A\x220.7800159454345703\x22,\x22xGChain\x22\x3A\x224.666777674108744\x22,\x22xGBuildup\x22\x3A\x224.53211510553956\x22\x7D,\x7B\x22id\x22\x3A\x222148\x22,\x22player_name\x22\x3A\x22Joaqu\x5Cu00edn\x22,\x22games\x22\x3A\x2217\x22,\x22time\x22\x3A\x22343\x22,\x22goals\x22\x3A\x220\x22,\x22xG\x22\x3A\x221.027239991351962\x22,\x22assists\x22\x3A\x221\x22,\x22xA\x22\x3A\x221.3191349804401398\x22,\x22shots\x22\x3A\x226\x22,\x22key_passes\x22\x3A\x2219\x22,\x22yellow_cards\x22\x3A\x222\x22,\x22red_cards\x22\x3A\x220\x22,\x22position\x22\x3A\x22M\x20S\x22,\x22team_title\x22\x3A\x22Real\x20Betis\x22,\x22npg\x22\x3A\x220\x22,\x22npxG\x22\x3A\x221.027239991351962\x22,\x22xGChain\x22\x3A\x223.506589997559786\x22,\x22xGBuildup\x22\x3A\x222.1130624152719975\x22\x7D,\x7B\x22id\x22\x3A\x222168\x22,\x22player_name\x22\x3A\x22Beb\x5Cu00e9\x22,\x22games\x22\x3A\x224\x22,\x22time\x22\x3A\x2251\x22,\x22goals\x22\x3A\x220\x22,\x22xG\x22\x3A\x220.05082689970731735\x22,\x22assis

>Solution :

You get that text basically because the website uses a JavaScript Object. You have to extract the data from the JS object. Try this:


script_tag = content.find("script", text=re.compile("playersData"))

json_data = re.search("playersData\s+=\s+JSON.parse\('([^']+)", script_tag.string).group(1)

json_data = json_data.encode("utf-8").decode("unicode_escape")
players_data = json.loads(json_data)

# Print the extracted data
print(players_data)