How to scrape text outside of href with BeautifulSoup

May 17, 2022

I’m trying to scrape the text "Woodford Reserve Master Collection Five Malt Stouted Mash" from the following:

<a class="catalog_item_name" aria-hidden="true" tabindex="-1" id="WC_CatalogEntryDBThumbnailDisplayJSPF_3074457345616901168_link_9b" href="/webapp/wcs/stores/servlet/ProductDisplay?catalogId=10051&storeId=10051&productId=3074457345616901168&langId=-1&partNumber=000086630prod&errorViewName=ProductDisplayErrorView&categoryId=1334014&top_category=25208&parent_category_rn=1334013&urlLangId=&variety=American+Whiskey&categoryType=Spirits&fromURL=%2fwebapp%2fwcs%2fstores%2fservlet%2fCatalogSearchResultView%3fstoreId%3d10051%26catalogId%3d10051%26langId%3d-1%26categoryId%3d1334014%26variety%3dAmerican%2bWhiskey%26categoryType%3dSpirits%26top_category%3d%26parent_category_rn%3d%26sortBy%3d5%26searchSource%3dE%26pageView%3d%26beginIndex%3d">Woodford Reserve Master Collection Five Malt Stouted Mash</a>

I am able to scrape the href using the following code, however can’t seem to be able to scrape the title text separately:


for product in soup.select('a.catalog_item_name'):
    link.append(product['href'])

print(link)

I have also tried

for product in soup.select('a.catalog_item_name'):
    link.append(product.a['href'])

print(link)

However I can’t seem to quite capture the title information separately. Thanks in advance for the help!

>Solution :

Try:

data=[]
for product in soup.select('a.catalog_item_name'):
    link=product['href']
    title=product.get_text()
    data.append([link,title])

print(data)