How to extract table from webpage that requires click/toggle?

March 30, 2024

I’m trying to extract tables from this webpage, but am only able to get the pitching table for example. I want to get the hitting table as well, which would in theory be this URL:
https://www.covers.com/sport/baseball/mlb/matchup/279605#hitting

But that doesn’t get me any different result. Separately I want to be able to extract things like the Last 5 and Last 10 options, but I assume the answer is going to be same for all of these. How would I go about setting this up so it could dynamically select and extract these tables?

import pandas as pd
import requests
from bs4 import BeautifulSoup

url = 'https://www.covers.com/sport/baseball/mlb/matchup/279605'

pd.set_option('display.max_rows', 5000)
pd.set_option('display.max_columns', 5000)
pd.set_option('display.width', 1500)

html = requests.get(url).content   
df_list = pd.read_html(html)
df = df_list[12]
print(df)

>Solution :

The hitting table is loaded from different URL. You can use this example to load it:

import pandas as pd

hitting_table_url = "https://www.covers.com/sport/baseball/mlb/matchup/279605/stats-analysis/hitting/overall"

df = pd.read_html(hitting_table_url)[0]
print(df)

Prints:

       Team  Runs/9   AVG.  Hits   HR   BB  EB Hits    OBP
0  Colorado     4.5  0.246   8.4  1.0  2.7      1.0  0.309
1   Arizona     4.7  0.248   8.5  1.1  3.3      1.1  0.320