Webscraper Python Beautifulsoup

December 17, 2021

I know what I’m trying to do is the simplest but it blows my mind. I’d like pull data from HTML page (https://partner.microsoft.com/en-us/membership/application-development-competency) using BeautifulSoup. To do that I need to use the .find() function I guess. Ain’t know what to do no more. Appreciate every form of help.
Here’s the HTML I’m working with:
[enter image description here][1]
[1]: https://i.stack.imgur.com/sHAMF.png

import requests
from bs4 import BeautifulSoup 

url = 'https://partner.microsoft.com/en-us/membership/application-development-competency'
res = requests.get(url)
html_page = res.content
soup = BeautifulSoup(html_page, 'html.parser')
text = soup.find("div",{"class":"col-md4[2]"})

output = ''
blacklist = [
    'style',
    'head',
    'meta',
    'col-md4[0]',
    'col-md4[1]',
]

for t in text:
    if t.parent.name not in blacklist:
        output += '{} '.format(t)

    sheet = '<html><body>' + text + '</body></html>';
    file_object  = open("record.html", "w+");
    file_object.write(sheet);
    file_object.close();

[enter image description here][1]
[1]: https://i.stack.imgur.com/sHAMF.png

>Solution :

You can do something like this, but for this site i think better use XPath

import requests
from bs4 import BeautifulSoup


url = 'https://partner.microsoft.com/en-us/membership/application-development-competency'
response = requests.get(url)
soup = BeautifulSoup(response.text, features="lxml")
cols = soup.find_all("div", class_="col-md-4")
print(cols[6].getText())