Hello all I am looking to scrape the leaderboard table similar to the one @chitown88 did in Having Trouble Clicking in Date Field with Selenium
The link is https://rotogrinders.com/resultsdb/nfl for the date 02/13/2022.
>Solution :
It’s just a matter of finding the right url and then parsing it:
import requests
import pandas as pd
url = 'https://service.fantasylabs.com/live-contests/?sport=NFL&contest_group_id=71194'
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36'}
jsonData = requests.get(url, headers=headers).json()
contestIds = {}
for each in jsonData['live_contests']:
contestIds[each['contest_name']] = each['contest_id']
rows = []
for contestName, contestId in contestIds.items():
url = f'https://dh5nxc6yx3kwy.cloudfront.net/contests/nfl/20220213/{contestId}/lineups/'
jsonData = requests.get(url, headers=headers).json()
lineups = jsonData['lineups']
for k, v in lineups.items():
v.update({'contestName':contestName})
rows.append(v)
print(contestName)
df = pd.DataFrame(rows)
Output:
print(df.head().to_string())
lineupHash lineupCt lineupUserCt lineupPlayers points totalSalary totalOwn minOwn maxOwn avgOwn lineupRank isCashing favoriteCt underdogCt homeCt visitorCt payout lineupPercentile correlatedPlayers teamStacks gameStacks lineupTrends entryNameList contestName
0 6957:33786:1343:8491:8469:47456 1 1 {'CPT1': 6957, 'FLEX1': 33786, 'FLEX2': 1343, 'FLEX3': 8491, 'FLEX4': 8469, 'FLEX5': 47456} 97.98 44400 172.73 0.0 74.33 28.79 199517 False 5 1 0 0 0.0 42.51 6 {'2203': ['6957:1', '33786:0', '1343:0', '8491:0', '47456:0']} {'5229230': [6957, 33786, 1343, 8491, 8469, 47456]} {'qbPairedWithPassCatcher': True, 'qbStackPairedWithOpponent': False, 'rbPairedWithDefense': True, 'noPlayersAgainstDEF': False, 'maxOneRBPerGame': True, 'minOnePlayerWithLowOwnership': True, 'cptQBPairedWithMultiplePassCatchers': True, 'cptPassCatcherPairedWithQB': False, 'maxOneDEFPerGame': False, 'maxOneKickerPerGame': True} [Wendy318] NFL Showdown $6M Super Bowl LVI Millionaire [$1M to 1st] (Super Bowl)
1 8491:33786:6957:64586:1343:47456 143 140 {'CPT1': 8491, 'FLEX1': 33786, 'FLEX2': 6957, 'FLEX3': 64586, 'FLEX4': 1343, 'FLEX5': 47456} 99.32 47900 230.82 0.0 74.33 38.47 185392 False 5 1 0 0 0.0 39.50 6 {'2203': ['8491:1', '33786:0', '6957:0', '1343:0', '47456:0']} {'5229230': [8491, 33786, 6957, 64586, 1343, 47456]} {'qbPairedWithPassCatcher': True, 'qbStackPairedWithOpponent': True, 'rbPairedWithDefense': True, 'noPlayersAgainstDEF': False, 'maxOneRBPerGame': True, 'minOnePlayerWithLowOwnership': True, 'cptQBPairedWithMultiplePassCatchers': False, 'cptPassCatcherPairedWithQB': False, 'maxOneDEFPerGame': True, 'maxOneKickerPerGame': True} [dman16384, weezywe34, tradinfool, Ellio31, AshleeLing, thop78, yabo22, bmo115, tadpole453, jeffthemoron, parlaygods, deathgrip, KingDeer, vassassin, miamimariok, CoolTreez, MrWentz, tkeating11, DUB5588, bkat26, tankitytank, TheRooneyRule, BenSeligman, viktor122000, astros210, BATMAN1973, bastavros, tj620, dbuermann, briand83, magic253792, bobboman11, arbitrage88, tdvrbas, p0l4r21, Kbosh1989, dfecko12, beavea, invsbltorsoman, Ponta01, Francis1959, joseph3322, teeeeej, Eduardo17Gonzalez20, mme9482, 12strikesguy, Sturdy36, jhender, Mogag, MDTigers8, BobbyJ33, gabsdad2003, funfootball, mattblack70, jberry0461, ajhall03, plumbnasty361, bfahey1989, Benriggins86, cjcook017, losski1976, spider710, boosted23, ladodger13, zdub2344, furey17, TC2480, elctrichomes87, EuBaAsta, JeffHampe, lucascw10, hoffy555, jkaelin, williec7, Latrinadraft, MTtires, UncleBanjo, joejoe09, grrrrmonsta, jddez64, Tee247, tjw2004, yun.seungdo, KartoshkaIstanbul, exmo, insane1ac, bruins06, Gatorsjj40, JRCLCC, cmitc053, dmask9, Butts11, brielly1972, csh9022, lewishans1213, Troyalty14, Jaystar4lyfe, GigelB05, clayton_521, Jjluker, ...] NFL Showdown $6M Super Bowl LVI Millionaire [$1M to 1st] (Super Bowl)
2 8491:33786:6957:64586:1343:8469 494 465 {'CPT1': 8491, 'FLEX1': 33786, 'FLEX2': 6957, 'FLEX3': 64586, 'FLEX4': 1343, 'FLEX5': 8469} 97.32 49500 226.04 0.0 74.33 37.67 208970 False 4 2 0 0 0.0 44.52 6 {'2203': ['8491:1', '33786:0', '6957:0', '1343:0'], '2141': ['64586:0', '8469:0']} {'5229230': [8491, 33786, 6957, 64586, 1343, 8469]} {'qbPairedWithPassCatcher': True, 'qbStackPairedWithOpponent': True, 'rbPairedWithDefense': False, 'noPlayersAgainstDEF': False, 'maxOneRBPerGame': True, 'minOnePlayerWithLowOwnership': True, 'cptQBPairedWithMultiplePassCatchers': False, 'cptPassCatcherPairedWithQB': False, 'maxOneDEFPerGame': False, 'maxOneKickerPerGame': True} [cinodolla5, drew_streckenbach, jfin711, HappyHollow74, EuBaAsta, Gconstance88, kinvest0314, Coltron, bigbuck1995, PaulusCarolus, dredre69, jsmooth816, andrewkole, Mandawg7676, RedskinsUNCfan, IcemanBK, moeryp, benchristine1, tenthletta, gtully404, lpmurphy, TxFantasyStud, FOMOSummer2, big_ugh, jdmcconnell75, AlmightyZoe, medlet01, pjs1974, jrl2002, kdevon09, Tuckerjr007, Wentz2Ertz, CarterBoys2006, Chunkytwilight, heyo212, FalseProphetxX, brianr87, Charms03, ramster_329, milkchoc021, supe302, DraftSmasher15, jluvv30, lhutzler, gman240, KynRi, mcork1223, jval71, Pigbear, jack.ryan369, saffronsnake, lenster1988, pipes22, lboogystl, DimDarkness10, 2hot4u2, robfree28, Bigkat777, dabears2424, xGhostMachinex, Raekwon19, rickychurch, jmorreale3, Roll...Tide, warrior75, 1PerfectLineup, bronsonnemelka, 05Doran.0505, Score112, TimWinner, oakraidrluva26, TOMMYBBQ1, BigPawka, jeverh11, DTopps, BradChavez, jbean850, b.nason1224, miller1116, beekiewah18, vimvicious, mattyyoung1911, hauk1973, bradleycounts8, Suchislife, FdThree, MrFrodo444, leek44, hbfrank949, hammertime77, mon122690, dmilner100, mrmpls701, yellowpup60, MarcPerch, spop17, 7danan, Gman1972, fperanio, 00BallsDeep00, ...] NFL Showdown $6M Super Bowl LVI Millionaire [$1M to 1st] (Super Bowl)
3 1343:33786:6957:64586:8491:54513 85 82 {'CPT1': 1343, 'FLEX1': 33786, 'FLEX2': 6957, 'FLEX3': 64586, 'FLEX4': 8491, 'FLEX5': 54513} 97.12 49400 205.11 0.0 74.33 34.18 211429 False 5 1 0 0 0.0 45.05 6 {'2203': ['1343:1', '33786:0', '6957:0', '8491:0', '54513:0']} {'5229230': [1343, 33786, 6957, 64586, 8491, 54513]} {'qbPairedWithPassCatcher': True, 'qbStackPairedWithOpponent': True, 'rbPairedWithDefense': False, 'noPlayersAgainstDEF': False, 'maxOneRBPerGame': True, 'minOnePlayerWithLowOwnership': True, 'cptQBPairedWithMultiplePassCatchers': False, 'cptPassCatcherPairedWithQB': True, 'maxOneDEFPerGame': True, 'maxOneKickerPerGame': True} [iriestylee, millameterpeter, ihoagied, randallcopp, wager_32, msam, Gthitter, lpmurphy, retselwolsac, unclezeke, kokomo2433, Skibasher, babytempr17, philney, RoarinBoren, michaelchite, acetx555, robertnyg, cutemike37, malip1133, IamApollo68, Sparkle01, jreth24, OperationWalrus, mtw13002, Fedgret9199, andrewtbell, S.Cornett, farmboyz, markwillig, rhouston13, danieltmv, TreyDay, bkenard, mmalczyk, MVanSlooten, cxmele, jujack, tnguyen123, rkess100, aanderson61, brknets23, ahitspat, BryxAlv, BowieOrMaze, MasterDrafter007, MooreP7, DynamicGOB, rabblerouser, missbeccax3, baseballfan37, LlamaSan, cevansb0, anddgrah, mcgintymc, Twebba71, piledriver40, AnointedMatt, JJHawk30, RickyRococo, Ddohnalek, sport81, losingplace, kencamnat, Bashman1234, bigmac34786, KingOzzie, ctompkin, puckhead18, prettyboi90, kccputech, loegs14, KPerry6, Jordan993, clibass1, clivebixby, steve4booze, rjbiz123, ztrivett21, dnice39, GoGators1509, sevendrums] NFL Showdown $6M Super Bowl LVI Millionaire [$1M to 1st] (Super Bowl)
4 47456:33786:6957:64586:1343:8491 11 11 {'CPT1': 47456, 'FLEX1': 33786, 'FLEX2': 6957, 'FLEX3': 64586, 'FLEX4': 1343, 'FLEX5': 8491} 99.32 47000 243.65 0.0 74.33 40.61 185535 False 5 1 0 0 0.0 39.53 6 {'2203': ['47456:1', '33786:0', '6957:0', '1343:0', '8491:0']} {'5229230': [47456, 33786, 6957, 64586, 1343, 8491]} {'qbPairedWithPassCatcher': True, 'qbStackPairedWithOpponent': True, 'rbPairedWithDefense': True, 'noPlayersAgainstDEF': False, 'maxOneRBPerGame': True, 'minOnePlayerWithLowOwnership': True, 'cptQBPairedWithMultiplePassCatchers': False, 'cptPassCatcherPairedWithQB': False, 'maxOneDEFPerGame': True, 'maxOneKickerPerGame': True} [Hardfour, p0l4r21, twonine29, jaks13, boltscrazy, SkillGame84, Omriking, sapplesauce55, gleetghost77, sspencer06, jozzy13] NFL Showdown $6M Super Bowl LVI Millionaire [$1M to 1st] (Super Bowl)
....
[87031 rows x 24 columns]