How to fix 302 redirect Scrapy?

May 21, 2022

I am trying to scrape https://howlongtobeat.com, but I keep getting 302 redirects. I found that the website is using ajax from the network monitor.

My code:

class HltbSpider(scrapy.Spider):
    name = 'hltb'
    def start_requests(self):
        for i in list(range(1,2)):
            
            url = f'https://howlongtobeat.com/search_results?page={i}'
            payload = "queryString=&t=games&sorthead=popular&sortd=0&plat=&length_type=main&length_min=&length_max=&v=&f=&g=&detail=&randomize=0"
            headers = {
                "content-type":"application/x-www-form-urlencoded",
                "user-agent": "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Mobile Safari/537.36"
            }

            yield scrapy.Request(url, meta = {'dont_redirect': True,'handle_httpstatus_list': [302]}, method='POST', body=payload, headers=headers, callback=self.parse)

           

    def parse(self, response):
        cards = response.css('div[class="search_list_details"]')

        for card in cards: 
            game_name = card.css('a[class=text_white]::attr(title)').get()
            game_dict =  {"Game_name":game_name}
           
            yield game_dict

It was working previously when all of the sudden it stopped working and I kept getting the 302 redirects. What seems to be the issue?