Python Extract Specific Data from Webscraping

December 23, 2021

I’m trying to extract specific data from a webpage using this code:

from bs4 import BeautifulSoup
import requests

source = requests.get('https://www.dailyfx.com/sentiment-report').text

soup = BeautifulSoup(source, 'lxml')
#print(soup.prettify())
price = soup.find_all('span', {'class' : 'gsstx'})


print(price)

but it just prints out everything in the gsstx span class like this:

, Retail trader data shows 76.23% of traders are net-long with the ratio of traders long to short at 3.21 to 1. , We typically take a contrarian view to crowd sentiment, and the fact traders are net-long suggests USD/CHF prices may continue to fall. , Retail trader data shows 31.96% of traders are net-long with the ratio of traders short to long at 2.13 to 1. , We typically take a contrarian view to crowd sentiment, and the fact traders are net-short suggests USD/JPY prices may continue to rise. , **Retail trader data shows 44.10% of traders are net-long with the ratio of traders short to long at 1.27 to 1.** , We typically take a contrarian view to crowd sentiment, and the fact traders are net-short suggests Wall Street prices may continue to rise. ]

How can I just print out this part?

Retail trader data shows 44.10% of traders are net-long with the ratio of traders short to long at 1.27 to 1.

>Solution :

For extracting and saving in csv, use following code (SPECIFIC TO THIS USE CASE AS SKIP MACHANSIM WILL NOT BE SAME IN ALL CASES)

from bs4 import BeautifulSoup
import requests

source = requests.get('https://www.dailyfx.com/sentiment-report').text

soup = BeautifulSoup(source, 'lxml')
#print(soup.prettify())
price = soup.find_all('span', {'class' : 'gsstx', 'style':"font-weight:bold;"})
lis = []
skip = 0
for i in price:
    if skip%2==0:
        lis.append(i.get_text())
    skip+=1

df = pd.DataFrame({"Data":lis})
df.to_csv("Data.csv",index=False)