Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Split text at specific character in BeautifulSoup

I am brand new to Python and BeautifulSoup so please forgive the lack of proper vocabulary in my question.

I am trying to extract the list from this webpage: http://spajournalism.com/membership/ – I want all the publications that are asssociated with a specific university. I’d like to end up with a list of dictionaries like:
[{publication_url: url1, publication_name: name1, uni: uni1}, {publication_url: url2, publication_name: name2, uni: uni2}]

Unfortunately the content on the webpage is quite messy, HTML-wise and it’s proving tricky. My code is currently:

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

import lxml.etree
import requests
from bs4 import BeautifulSoup

url = "http://spajournalism.com/membership/"
page = requests.get(url)
soup = BeautifulSoup(page.content, "lxml")

section = soup.find("div", "entry-content clearfix")
links = section.find_all("a")

#list = []

#for link in links:
#    publication = {
#        "Link" : link.get("href"),
#        "Publication" : link.parent.text
#    }

for link in links:
    print("Link: ", link.get("href"), "Text: ", link.parent.text)

This returns a list of the following nature:

Link:  http://www.swanseastudentmedia.com/waterfront/ Text:  The Waterfront – Swansea University
Link:  https://www.seren.bangor.ac.uk/ Text:  Y Seren – Bangor University
...etc

I would like to, instead of getting all the text in one go with link.parent.text, split it at the hyphen ( – ), and get something more like:

Link:  http://www.swanseastudentmedia.com/waterfront/ Text:  The Waterfront University: Swansea University
Link:  https://www.seren.bangor.ac.uk/ Text:  Y Seren University: Bangor University
...etc

I have tried something like the following:

for link in links:
    text = link.parent.text
    linktext = link.string
    text.replace(linktext, " ") # Replace the redundant link text with nothing

    print("Link: ", link.get("href"), "Publication: ", linktext, "University: ", text)

But the replacing the redundant text with nothing doesn’t seem to work because what I get is:

Link:  http://www.swanseastudentmedia.com/waterfront/ Publication:  The Waterfront University:  The Waterfront – Swansea University
Link:  https://www.seren.bangor.ac.uk/ Publication:  Y Seren University:  Y Seren – Bangor University
...etc

Is there a way of doing this? Any searches I do are full of results referring to something called Dash which isn’t relevant to me. Thanks 🙂

>Solution :

replace() function dont change the text_variable, it returns a new string

Change

text.replace(linktext, " ")

To

text = text.replace(linktext, " ").split("–", 1)[-1].strip()
Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading