How to extract tweet from Twitter using Python Selenium

November 9, 2023

I have problems about extracting tweets from Twitter. As I extracting tweets with Python Selenium, after a while Twitter occurs "Can’t load page". To solve this, I wrote time.sleep() method after extract tweet. But after N times, it will occurs again. How can i solve it?

The code is below:

driver = webdriver.Chrome()

driver.get(URL)

while(counter != tweets_count):
        # Get tweets
        for i in range(1,6):
            context = driver.find_element(By.XPATH, "//*[@id='react-root']/div/div/div[2]/main/div/div/div/div[1]/div/div[3]/section/div/div/div[" + str(i) + "]")
            context_list = context.text.split("\n")
            context_list = context_list[4:-5]
            temp = ""

            for j in range(len(context_list)):
                temp += context_list[j]

            tweets.append(temp)
            
            time.sleep(3)
        
        # Scroll down to bottom
        driver.execute_script("window.scrollTo(" + str(first_height) + ", " + str(increase_amount) + ");")
        
        first_height = increase_amount
        increase_amount = increase_amount+increase_amount
        
        # Waiting until page loading completely
        time.sleep(7)
        
        counter += 1
        print(counter)

"Can’t load page!". But I want to expect N times works.