BeautifulSoup to get data from a nested table

January 29, 2022

I am trying to pull data from a nested table in html

I can get BeautifulSoup to get the other divs but can’t get it to see the table.

This is what I’ve got so far:

import requests
from bs4 import BeautifulSoup


url = ''
response = requests.get(url)

soup = BeautifulSoup(response.content, 'html.parser')
d = soup.find(td, class_='DT_Text DT_R_Align sorting_1)

I’ve tried numerous variations. Tried going after div classes, div ids, td classes. Nothing has worked.

Is there anyway to get this data?

Thanks

>Solution :

I just checked the Network tab of Firefox’s Dev Tools while loading the page and found that there’s a GET request to https://s3.amazonaws.com/datafusion.web.generac.can/Data/Generac_MTN_co.json which contains the data inside the table. Your code can be modified to the followings:

import requests
import json

response = requests.get("https://s3.amazonaws.com/datafusion.web.generac.can/Data/Generac_MTN_co.json")
data = json.loads(response.content)
print(data)

# {'currentdatetime': '1/29/2022 12:17:00 AM', 'outage': [{'Name': 'Alberta', 'OUT': 0, 'SRV': 0, 'SFIPS': '60'}, {'Name': 'British Columbia', 'OUT': 24, 'SRV': 0, 'SFIPS': '61'}, {'Name': 'Manitoba', 'OUT': 75, 'SRV': 0, 'SFIPS': '62'}, {'Name': 'New Brunswick', 'OUT': 283, 'SRV': 0, 'SFIPS': '63'}, {'Name': 'Nova Scotia', 'OUT': 12, 'SRV': 0, 'SFIPS': '65'}, {'Name': 'Ontario', 'OUT': 417, 'SRV': 0, 'SFIPS': '66'}, {'Name': 'Quebec', 'OUT': 342, 'SRV': 0, 'SFIPS': '68'}, {'Name': 'Saskatchewan', 'OUT': 0, 'SRV': 0, 'SFIPS': '69'}, {'Name': 'Newfoundland', 'OUT': 0, 'SRV': 0, 'SFIPS': '64'}, {'Name': 'Prince Edward Island', 'OUT': 2, 'SRV': 0, 'SFIPS': '67'}]}