Yes I assume there was a subtle change, but I'm not saavy enough to figure out where and how to fix. Here is an excerpt of my code, which maybe is clunky, I don't know. It was trial and error - and it worked before. I've removed some lines and details (i.e. note in the REQURL link -- I have a loop that goes through all the teams and weeks and inside that loop it creates the correct url for that team/week.)
#This URL will be the URL that your login form points to with the "action" tag.
POSTURL = "https://www.rtsports.com"
payload = {
'ACCOUNTID': '<ID removed>'
'PASSWORD': '<pwd removed>'
}
#This URL is the page you actually want to pull down with requests.
REQURL = 'https://www.rtsports.com/football/team-capsules.php?LID=23611&UID=rnj07n3via1124l&TID='+team_rt_dict[t]+'&FWK='+str(w)
with requests.Session() as session:
post = session.post(POSTURL, data=payload)
r = session.get(REQURL)
soup = BeautifulSoup(r.text, 'lxml') # Parse the HTML as a string
rt_table=soup.find('table',{'class':'table table-no-borders table-tight table-hover '})
rt_df_raw = pd.read_html(str(rt_table))
rt_df=rt_df_raw[0] #this is now the dataframe for one team and one week
It looks like it performs fine through the creation "rt_table" -- which is supposed to be the html just for the one table on the page I am interested in. Here is what rt_table looks like -- I snipped out the middle section, leaving just the first 2 player rows and the last 2 player rows:
(EDIT TO ADD: I think the data stored in "rt_table" is OK, because I took the below and pasted into a web-based html-to-excel converter and everything came out good -- all players with all their stats in rows with correct headers)
<table class="table table-no-borders table-tight table-hover "><thead><tr><th>PLAYER</th><th>POS</th><th>NFL</th><th>BYE</th><th>INJ</th><th>OPP</th><th>DATE</th><th>LINEUP</th><th>PTS</th><th>SCORING</th></tr></thead><tbody></tbody><tr class="PlayerRow"><td><a href="javascript:ShowPlayer(11189);">Ben Roethlisberger <span class="glyphicon glyphicon-volume-up"></span></a></td><td>QB</td><td>PIT</td><td class="text-center">4</td><td class="text-center"><span class="injury InjP" data-original-title="Probable - Quad" data-placement="top" data-toggle="tooltip">P</span></td><td class="text-center"><span class="small">@</span>NYG</td><td><a href="/football/nfl-live-boxscore.php?LID=23611&amp;UID=rnj07n3via1124l&amp;X=344288&amp;GC=20200914019">Final</a> | <a href="/football/syndicated-news.php?LID=23611&amp;UID=rnj07n3via1124l&amp;X=903586&amp;ART=20200914221524508495508">Recap</a></td><td class="text-center"><strong>Starter</strong></td><td class="text-right">23</td><td class="text-left" style="padding-left:15px;">Ben Roethlisberger 3 passing TDs (18 pts)<br/> . . . Ben Roethlisberger 10 yd TD pass to JuJu Smith-Schuster<br/> . . . Ben Roethlisberger 13 yd TD pass to James Washington<br/> . . . Ben Roethlisberger 8 yd TD pass to JuJu Smith-Schuster<br/>Ben Roethlisberger 229 passing yds (5 pts)<br/></td></tr><tr class="PlayerRow"><td><a href="javascript:ShowPlayer(16108);">Austin Ekeler <span class="glyphicon glyphicon-volume-up"></span></a></td><td>RB</td><td>LAC</td><td class="text-center">6</td><td class="text-center"><span class="injury InjX" data-original-title="On IR - hamstring" data-placement="top" data-toggle="tooltip">X</span></td><td class="text-center"><span class="small">@</span>CIN</td><td><a href="/football/nfl-live-boxscore.php?LID=23611&amp;UID=rnj07n3via1124l&amp;X=348565&amp;GC=20200913004">Final</a> | <a href="/football/syndicated-news.php?LID=23611&amp;UID=rnj07n3via1124l&amp;X=562403&amp;ART=20200913193545156385708">Recap</a></td><td class="text-center"><strong>Starter</strong></td><td class="text-right">4</td><td class="text-left" style="padding-left:15px;">Austin Ekeler 84 rushing yds (4 pts)<br/></td></tr>
<<<<<SNIPPED>>>>>>
<tr class="PlayerRow"><td><a href="javascript:ShowPlayer(16612);">Jace Sternberger</a></td><td>TE</td><td>GNB</td><td class="text-center">5</td><td class="text-center"></td><td class="text-center"><span class="small">@</span>MIN</td><td><a href="/football/nfl-live-boxscore.php?LID=23611&amp;UID=rnj07n3via1124l&amp;X=821168&amp;GC=20200913016">Final</a> | <a href="/football/syndicated-news.php?LID=23611&amp;UID=rnj07n3via1124l&amp;X=287930&amp;ART=20200913164324664323708">Recap</a></td><td class="text-center">Bench</td><td class="text-right">0</td><td class="text-left" style="padding-left:15px;"></td></tr><tr class="PlayerRow"><td><a href="javascript:ShowPlayer(15244);">C.J. Uzomah</a></td><td>TE</td><td>CIN</td><td class="text-center">9</td><td class="text-center"><span class="injury InjX" data-original-title="On IR - torn right Achilles" data-placement="top" data-toggle="tooltip">X</span></td><td class="text-center">LAC</td><td><a href="/football/nfl-live-boxscore.php?LID=23611&amp;UID=rnj07n3via1124l&amp;X=996392&amp;GC=20200913004">Final</a> | <a href="/football/syndicated-news.php?LID=23611&amp;UID=rnj07n3via1124l&amp;X=195296&amp;ART=20200913193545156385708">Recap</a></td><td class="text-center">Bench</td><td class="text-right">4</td><td class="text-left" style="padding-left:15px;">C.J. Uzomah 45 receiving yds (4 pts)<br/></td></tr></table>
but, rt_df_raw only pulls in the column headers...which look fine, its just that the dataset itself is empty:
Code:
rt_df_raw
[Empty DataFrame
Columns: [PLAYER, POS, NFL, BYE, INJ, OPP, DATE, LINEUP, PTS, SCORING]
Index: []]