BeautifulSoup: Extracting a Title and adjacent <a> tags

December 8, 2021

I’m attempting to get data from Wikipedias sidebar on the ‘Current Events’ page with the below. At the moment this produces an array of Objects each with value title and url.

I would also like to provide a new value to the objects in array headline derived from the <h3> id or text content. This would result in each object having three values: headline, url and title. However, I’m unsure how to iterate through these.

Beautiful Soup Code

        soup = BeautifulSoup(response, "html.parser").find('div', {'aria-labelledby': 'Ongoing_events'})
        links = soup.findAll('a')


        for item in links:
            title = item.text
            url = ("https://en.wikipedia.org"+item['href'])

            eo = CurrentEventsObject(title, url)
            eventsArray.append(eo)

Wikipedia Current Events List

<div class="mw-collapsible-content">
    <h3><span class="mw-headline" id="Disasters">Disasters</span</h3>
    <ul>
        <li><a href="/wiki/Climate_crisis" title="Climate crisis">Climate crisis</a></li>
        <li><a href="/wiki/COVID-19_pandemic" title="COVID-19 pandemic">COVID-19 pandemic</a></li>
        <li><a href="/wiki/2021%E2%80%9322_European_windstorm_season" title="2021–22 European windstorm season">2021–22 European windstorm season</a></li>
        <li><a href="/wiki/2020%E2%80%9321_H5N8_outbreak" class="mw-redirect" title="2020–21 H5N8 outbreak">2020–21 H5N8 outbreak</a></li>
        <li><a href="/wiki/2021_Pacific_typhoon_season" title="2021 Pacific typhoon season">2021 Pacific typhoon season</a></li>
        <li><a href="/wiki/2021_Madagascar_food_crisis" class="mw-redirect" title="2021 Madagascar food crisis">Madagascar food crisis</a></li>
        <li><a href="/wiki/Water_scarcity_in_Iran" title="Water scarcity in Iran">Water crisis in Iran</a></li>
        <li><a href="/wiki/Famine_in_Yemen_(2016%E2%80%93present)" title="Famine in Yemen (2016–present)">Yemeni famine</a></li>
        <li><a href="/wiki/2021_Cumbre_Vieja_volcanic_eruption" title="2021 Cumbre Vieja volcanic eruption">2021 La Palma eruption</a></li>
    </ul>
    <h3><span class="mw-headline" id="Economic">Economic</span></h3>
    <ul>
        <li><a href="/wiki/2020%E2%80%932021_global_chip_shortage" title="2020–2021 global chip shortage">2020–2021 global chip shortage</a></li>
        <li><a href="/wiki/2021_global_supply_chain_crisis" title="2021 global supply chain crisis">2021 global supply chain crisis</a></li>
        <li><a href="/wiki/COVID-19_recession" title="COVID-19 recession">COVID-19 recession</a></li>
        <li><a href="/wiki/Lebanese_liquidity_crisis" title="Lebanese liquidity crisis">Lebanese liquidity crisis</a></li>
        <li><a href="/wiki/Pandora_Papers" title="Pandora Papers">Pandora Papers leak</a></li>
        <li><a href="/wiki/2021_Sri_Lankan_economic_crisis" title="2021 Sri Lankan economic crisis">Sri Lankan economic and food crisis</a></li>
        <li><a href="/wiki/2018%E2%80%932021_Turkish_currency_and_debt_crisis" title="2018–2021 Turkish currency and debt crisis">Turkish currency and debt crisis</a></li>
        <li><a href="/wiki/2021_United_Kingdom_natural_gas_supplier_crisis" title="2021 United Kingdom natural gas supplier crisis">United Kingdom natural gas supplier crisis</a></li>
    </ul>
    <h3><span class="mw-headline" id="Politics">Politics</span></h3>
    <ul>
        <li><a href="/wiki/2021_Belarus%E2%80%93European_Union_border_crisis" title="2021 Belarus–European Union border crisis">Belarus−European Union border crisis</a></li>
        <li><a href="/wiki/2021_Brazilian_protests" title="2021 Brazilian protests">Brazilian protests</a></li>
        <li><a href="/wiki/2021_Colombian_protests" title="2021 Colombian protests">Colombian tax reform protests</a></li>
        <li><a href="/wiki/2021_Eswatini_protests" title="2021 Eswatini protests">Eswatini protests</a></li>
        <li><a href="/wiki/2018%E2%80%932021_Haitian_protests" title="2018–2021 Haitian protests">Haitian protests</a></li>
        <li><a href="/wiki/2020%E2%80%932021_Indian_farmers%27_protest" title="2020–2021 Indian farmers' protest">Indian farmers' protests</a></li>
        <li><a href="/wiki/Insulate_Britain_protests" title="Insulate Britain protests">Insulate Britain protests</a></li>
        <li><a href="/wiki/2021_Jersey_dispute" title="2021 Jersey dispute">Jersey dispute</a></li>
        <li><a href="/wiki/Libyan_peace_process" title="Libyan peace process">Libyan peace process</a></li>
        <li><a href="/wiki/2020%E2%80%9321_Malaysian_political_crisis" title="2020–21 Malaysian political crisis">Malaysian political crisis</a></li>
        <li><a href="/wiki/2021_Myanmar_protests" title="2021 Myanmar protests">Myanmar protests</a></li>
        <li><a href="/wiki/2018%E2%80%932021_Nicaraguan_protests" title="2018–2021 Nicaraguan protests">Nicaraguan protests</a></li>
        <li><a href="/wiki/End_SARS" title="End SARS">Nigerian protests</a></li>
        <li><a href="/wiki/2019%E2%80%932021_Persian_Gulf_crisis" title="2019–2021 Persian Gulf crisis">Persian Gulf crisis</a></li>
        <li><a href="/wiki/2017%E2%80%93present_Peruvian_political_crisis" title="2017–present Peruvian political crisis">Peruvian crisis</a></li>
        <li><a href="/wiki/2021_Russian_election_protests" title="2021 Russian election protests">Russian election protests</a></li>
        <li><a href="/wiki/2021_Solomon_Islands_unrest" title="2021 Solomon Islands unrest">Solomon Islands unrest</a></li>
        <li><a href="/wiki/Tigrayan_peace_process" title="Tigrayan peace process">Tigrayan peace process</a></li>
        <li><a href="/wiki/2020%E2%80%932021_Thai_protests" title="2020–2021 Thai protests">Thai protests</a></li>
        <li><a href="/wiki/2021_Tunisian_political_crisis" title="2021 Tunisian political crisis">Tunisian political crisis</a></li>
        <li><a href="/wiki/2020%E2%80%932021_United_States_racial_unrest" title="2020–2021 United States racial unrest">United States racial unrest</a></li>
        <li><a href="/wiki/Venezuelan_presidential_crisis" title="Venezuelan presidential crisis">Venezuelan presidential crisis</a></li>
    </ul>
    <div class="editlink noprint plainlinks"><a class="external text" href="https://en.wikipedia.org/w/index.php?title=Portal:Current_events/Sidebar&amp;action=edit">edit section</a></div>
</div>

>Solution :

Note: Try to select your elements more specific to get all information in one process – Defining a list outside your loops will avoid from overwriting

Following steps will create a list of dicts, that for example could simply iterated or turned into a data frame.

#1
Select all <ul> that are direct siblings of a <h3>

soup.select('h3 + ul')

#2 Select the <h3> and get its text:

e.find_previous_sibling('h3').text.strip()

#3 Select all <a> in the <ul> and iterat the results while creating a list of dicts:

for a in e.select('a'):
    data.append({
        'headline':headline,
        'title': a['title'],
        'url':'https://en.wikipedia.org'+a['href']
})

Example

soup = BeautifulSoup(response, "html.parser").find('div', {'aria-labelledby': 'Ongoing_events'})

data = []

for e in soup.select('h3 + ul'):
    headline = e.find_previous_sibling('h3').text.strip()
    for a in e.select('a'):
        data.append({
            'headline':headline,
            'title': a['title'],
            'url':'https://en.wikipedia.org'+a['href']
        })

data

Output

[{'headline': 'Disasters',
  'title': 'Climate crisis',
  'url': 'https://en.wikipedia.org/wiki/Climate_crisis'},
 {'headline': 'Disasters',
  'title': 'COVID-19 pandemic',
  'url': 'https://en.wikipedia.org/wiki/COVID-19_pandemic'},
 {'headline': 'Disasters',
  'title': '2021–22 European windstorm season',
  'url': 'https://en.wikipedia.org/wiki/2021%E2%80%9322_European_windstorm_season'},
 {'headline': 'Disasters',
  'title': '2020–21 H5N8 outbreak',
  'url': 'https://en.wikipedia.org/wiki/2020%E2%80%9321_H5N8_outbreak'},
 {'headline': 'Disasters',
  'title': '2021 Pacific typhoon season',
  'url': 'https://en.wikipedia.org/wiki/2021_Pacific_typhoon_season'},
 {'headline': 'Disasters',
  'title': '2021 Madagascar food crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_Madagascar_food_crisis'},
 {'headline': 'Disasters',
  'title': 'Water scarcity in Iran',
  'url': 'https://en.wikipedia.org/wiki/Water_scarcity_in_Iran'},
 {'headline': 'Disasters',
  'title': 'Famine in Yemen (2016–present)',
  'url': 'https://en.wikipedia.org/wiki/Famine_in_Yemen_(2016%E2%80%93present)'},
 {'headline': 'Disasters',
  'title': '2021 Cumbre Vieja volcanic eruption',
  'url': 'https://en.wikipedia.org/wiki/2021_Cumbre_Vieja_volcanic_eruption'},
 {'headline': 'Economic',
  'title': '2020–2021 global chip shortage',
  'url': 'https://en.wikipedia.org/wiki/2020%E2%80%932021_global_chip_shortage'},
 {'headline': 'Economic',
  'title': '2021 global supply chain crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_global_supply_chain_crisis'},
 {'headline': 'Economic',
  'title': 'COVID-19 recession',
  'url': 'https://en.wikipedia.org/wiki/COVID-19_recession'},
 {'headline': 'Economic',
  'title': 'Lebanese liquidity crisis',
  'url': 'https://en.wikipedia.org/wiki/Lebanese_liquidity_crisis'},
 {'headline': 'Economic',
  'title': 'Pandora Papers',
  'url': 'https://en.wikipedia.org/wiki/Pandora_Papers'},
 {'headline': 'Economic',
  'title': '2021 Sri Lankan economic crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_Sri_Lankan_economic_crisis'},
 {'headline': 'Economic',
  'title': '2018–2021 Turkish currency and debt crisis',
  'url': 'https://en.wikipedia.org/wiki/2018%E2%80%932021_Turkish_currency_and_debt_crisis'},
 {'headline': 'Economic',
  'title': '2021 United Kingdom natural gas supplier crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_United_Kingdom_natural_gas_supplier_crisis'},
 {'headline': 'Politics',
  'title': '2021 Belarus–European Union border crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_Belarus%E2%80%93European_Union_border_crisis'},
 {'headline': 'Politics',
  'title': '2021 Brazilian protests',
  'url': 'https://en.wikipedia.org/wiki/2021_Brazilian_protests'},
 {'headline': 'Politics',
  'title': '2021 Colombian protests',
  'url': 'https://en.wikipedia.org/wiki/2021_Colombian_protests'},
 {'headline': 'Politics',
  'title': '2021 Eswatini protests',
  'url': 'https://en.wikipedia.org/wiki/2021_Eswatini_protests'},
 {'headline': 'Politics',
  'title': '2018–2021 Haitian protests',
  'url': 'https://en.wikipedia.org/wiki/2018%E2%80%932021_Haitian_protests'},
 {'headline': 'Politics',
  'title': "2020–2021 Indian farmers' protest",
  'url': 'https://en.wikipedia.org/wiki/2020%E2%80%932021_Indian_farmers%27_protest'},
 {'headline': 'Politics',
  'title': 'Insulate Britain protests',
  'url': 'https://en.wikipedia.org/wiki/Insulate_Britain_protests'},
 {'headline': 'Politics',
  'title': '2021 Jersey dispute',
  'url': 'https://en.wikipedia.org/wiki/2021_Jersey_dispute'},
 {'headline': 'Politics',
  'title': 'Libyan peace process',
  'url': 'https://en.wikipedia.org/wiki/Libyan_peace_process'},
 {'headline': 'Politics',
  'title': '2020–21 Malaysian political crisis',
  'url': 'https://en.wikipedia.org/wiki/2020%E2%80%9321_Malaysian_political_crisis'},
 {'headline': 'Politics',
  'title': '2021 Myanmar protests',
  'url': 'https://en.wikipedia.org/wiki/2021_Myanmar_protests'},
 {'headline': 'Politics',
  'title': '2018–2021 Nicaraguan protests',
  'url': 'https://en.wikipedia.org/wiki/2018%E2%80%932021_Nicaraguan_protests'},
 {'headline': 'Politics',
  'title': 'End SARS',
  'url': 'https://en.wikipedia.org/wiki/End_SARS'},
 {'headline': 'Politics',
  'title': '2019–2021 Persian Gulf crisis',
  'url': 'https://en.wikipedia.org/wiki/2019%E2%80%932021_Persian_Gulf_crisis'},
 {'headline': 'Politics',
  'title': '2017–present Peruvian political crisis',
  'url': 'https://en.wikipedia.org/wiki/2017%E2%80%93present_Peruvian_political_crisis'},
 {'headline': 'Politics',
  'title': '2021 Russian election protests',
  'url': 'https://en.wikipedia.org/wiki/2021_Russian_election_protests'},
 {'headline': 'Politics',
  'title': '2021 Solomon Islands unrest',
  'url': 'https://en.wikipedia.org/wiki/2021_Solomon_Islands_unrest'},
 {'headline': 'Politics',
  'title': 'Tigrayan peace process',
  'url': 'https://en.wikipedia.org/wiki/Tigrayan_peace_process'},
 {'headline': 'Politics',
  'title': '2020–2021 Thai protests',
  'url': 'https://en.wikipedia.org/wiki/2020%E2%80%932021_Thai_protests'},
 {'headline': 'Politics',
  'title': '2021 Tunisian political crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_Tunisian_political_crisis'},
 {'headline': 'Politics',
  'title': '2020–2021 United States racial unrest',
  'url': 'https://en.wikipedia.org/wiki/2020%E2%80%932021_United_States_racial_unrest'},
 {'headline': 'Politics',
  'title': 'Venezuelan presidential crisis',
  'url': 'https://en.wikipedia.org/wiki/Venezuelan_presidential_crisis'}]