Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Follow
Facebook Twitter Instagram Youtube
Contact

Can't scrape information going back to the number of days I specify

byMR
October 7, 2022

I want to grab the posting date and title from this webpage using a python script that I’ve already created. The script is working errorlessly.

Now I want to add some logic to the script so that it will only fetch stuff up to the day when I enter a number of days into variable days_to_go_back. For example, if I enter 2 right next to days_to_go_back, the script should scrape content up to October 6.

import requests
from bs4 import BeautifulSoup

link = 'https://atlanta.craigslist.org/search/lab'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
}

days_to_go_back = 2

with requests.Session() as s:
    s.headers.update(headers)
    res = s.get(link)
    soup = BeautifulSoup(res.text,"lxml")
    for item in soup.select("[id^='search-results'] li[class*='result']"):
        date = ' '.join(item.select_one("time.result-date").text.split())
        title = item.select_one("h3.result-heading > a").get_text(strip=True)
        print(date,title)

Current output:

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel 

Oct 7 General labor crew member needed
Oct 7 General Labor Demo
Oct 7 LAWN CARE & AERATION SPECIALISTS (ROSWELL, GA)
Oct 7 LAWN CARE & AERATION SPECIALISTS (ROSWELL, GA)
Oct 7 Craftsman Wanted
Oct 7 Home Cleaners Wanted! Make over $145+ for 1bd, $165 for 2bd cleans.
Oct 7 Sign Company In Search of Installers and Fabricators
Oct 7 ***PILE DRIVERS-DRILLERS-TRAINEES***
Oct 7 Warehouse Package Handler
Oct 7 Workers Needed for Local Fair (Cash Paid Daily)
Oct 7 Warehouse Team Member - Multiple Shifts Available
Oct 7 Construction
Oct 7 Lyft Driver (part-time / full-time)
Oct 7 Lyft Driver (part-time / full-time)
Oct 7 NEED HANDYMAN FOR TASKS - at least $35/hr, Paid Daily
Oct 7 Warehouse Team Member - $3,000 Sign-On Bonus
Oct 7 Warehouse Team Member - $3,000 Sign-On Bonus
Oct 7 Amazon Warehouse Assistant - Immediate Hire
Oct 7 Amazon Picker - Overnight Shifts Available
Oct 7 Warehouse Team Member - With Sign-On Bonus!
Oct 7 Amazon Warehouse Assistant
Oct 7 Need a Crew of 4+
Oct 6 Helper and Construction
Oct 6 Truck Driver/El Camionero
Oct 6 Workers Needed for Local Fair (Cash Paid Daily)
Oct 6 Roofing Crews Needed! ($2,000 Sign On Bonus - Start Today!)
Oct 6 Available Immediately General Labor/Driver with Truck or Van Positions
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 Automotive Technicians Needed
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 Automotive Technicians Needed
Oct 6 Automotive Technicians Needed
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 NEED HANDYMAN FOR TASKS - at least $35/hr, Paid Daily
Oct 6 $20-25/hr. SIMPLE VALET TRASH COLLECTOR-Camp Creek/East Point 30331
Oct 6 Asphalt workers labors
Oct 6 CDL-A Company Drivers - $5K SIGN-ON BONUS! 💰 🚚
Oct 6 EXPERIENCED HANDYMAN ***2 immediate openings
Oct 6 Automotive Detail /Porter/Shop Help
Oct 6 Fork Lift Operator/ Order Picker
Oct 6 Lot Operations
Oct 6 Waterproofing Foreman
Oct 6 Auto Detailer
Oct 6 Lyft Driver (part-time / full-time)
Oct 6 Lyft Driver (part-time / full-time)
Oct 6 FULL TIME WITH PAID BENEFITS $16+ HOUR
Oct 5 Georgia Canvasser
Oct 5 💚 CLEANING JOBS AVAILABLE 💚  $22+/hr, Paid Daily 💚
Oct 5 Farm Help Animal Care
Oct 5 Warehouse Team Member - $3,000 Sign-On Bonus
Oct 5 Maintenance/Custodial
Oct 5 Hiring House Keeper For Airbnb
Oct 5 Warehouse Material Handler: $15.92/hr,M-F/Full Time/Day Shift/Benefits
Oct 5 Laborers, Forman, and Drivers
Oct 5 NEED HANDYMAN FOR TASKS - at least $35/hr, Paid Daily
Oct 5 CANVASS POSITION (MAKE MONEY THIS WEEK)
Oct 5 Hiring full Auto Detailer
Oct 5 Amazon Packer - Part-time Night Shifts Available
Oct 5 Warehouse Team Member Sun - Wed (06:30PM - 05:00AM)
Oct 5 Amazon Warehouse Assistant - Immediate Hire
Oct 5 EC3 Construction Is Now Hiring Framers
Oct 5 Detailer Wanted $16.00hr. plus 250.00 BONUS$$$
Oct 5 FREE $10 CASH FOR EVERYONE GET IT TODAY! SUPPLIES LIMITED HURRY
Oct 5 Now Hiring: Diesel technicians
Oct 5 Warehouse Associate
Oct 5 Delivery Driver Needed
Oct 5 Drivers Helper - Weekly Pay
Oct 5 Landscape Crew Member
Oct 5 Seasonal Laborers
Oct 5 Seasonal Laborers
Oct 5 WINDOW AND DOOR INSTALLERS NEEDED
Oct 5 Commercial Roofers and Helpers
Oct 5 Lyft Driver (part-time / full-time)
Oct 5 Lyft Driver (part-time / full-time)
Oct 5 Warehouse Associate/Driver
Oct 5 Grand Opening! Now Hiring! Immediate work!
Oct 5 C.D.L Drivers wanted
Oct 5 SET UP Workers Needed for Local Fair (Cash Paid Daily)
Oct 5 Workers Needed for Local Fair (Cash Paid Daily)
Oct 4 Distribution Center General Worker Position
Oct 4 Distribution Center General Worker Position
Oct 4 Distribution Center General Worker Position
Oct 4 NEED HANDYMAN FOR TASKS - at least $35/hr, Paid Daily
Oct 4 Warehouse Worker
Oct 4 House Cleaner Needed
Oct 4 Janitor Job offer
Oct 4 HIRING TIRE TECHNICIANS
Oct 4 CDL-A Company Drivers - $5K SIGN-ON BONUS! 💰 🚚
Oct 4 Part Time Groundskeeper/Cleaner
Oct 4 Driver and helper
Oct 4 Full-Time Community Trash Collector
Oct 4 * Deliver with DoorDash *
Oct 4 Experienced maintenance men
Oct 4 Maintenance Manager for Luxury Apts!
Oct 4 Maintenance Manager for Luxury Apts!
Oct 4 Maintenance Tech II for Luxury Apts!
Oct 4 Maintenance Tech II for Luxury Apts!
Oct 4 Maintenance Tech I for Luxury Apts!
Oct 4 Stucco Repair Help
Oct 4 Lyft Driver (part-time / full-time)
Oct 4 Lyft Driver (part-time / full-time)
Oct 4 Workers Needed for Local Fair (Cash Paid Daily)
Oct 4 Warehouse Team Member - Multiple Shifts Available
Oct 4 🌊 Tidal Wave Lawrenceville ǀ Car Wash Attendant 🌊
Oct 4 🌊 Tidal Wave Kennesaw ǀ Car Wash Attendant 🌊
Oct 4 🌊 Tidal Wave Gainesville ǀ Car Wash Attendant 🌊
Oct 4 🌊 Tidal Wave Dacula ǀ Car Wash Attendant 🌊
Oct 4 🌊 Tidal Wave Canton ǀ Car Wash Attendant 🌊
Oct 4 Heavy equipment operator
Oct 4 **Full Time Building Cleaner (pressure washing/brush scrubbing) Needed**
Oct 4 Amazon Picker - Up to $16.5/hr
Oct 4 Warehouse Laborer - Immediate Hire
Oct 4 Warehouse Team Member - Split Shifts (20hr/week)
Oct 4 Warehouse Team Member - Morning Shifts Available
Oct 4 Warehouse Team Member - Multiple Shifts Available

Expected output (up to oct 6):

Oct 7 General labor crew member needed
Oct 7 General Labor Demo
Oct 7 LAWN CARE & AERATION SPECIALISTS (ROSWELL, GA)
Oct 7 LAWN CARE & AERATION SPECIALISTS (ROSWELL, GA)
Oct 7 Craftsman Wanted
Oct 7 Home Cleaners Wanted! Make over $145+ for 1bd, $165 for 2bd cleans.
Oct 7 Sign Company In Search of Installers and Fabricators
Oct 7 ***PILE DRIVERS-DRILLERS-TRAINEES***
Oct 7 Warehouse Package Handler
Oct 7 Workers Needed for Local Fair (Cash Paid Daily)
Oct 7 Warehouse Team Member - Multiple Shifts Available
Oct 7 Construction
Oct 7 Lyft Driver (part-time / full-time)
Oct 7 Lyft Driver (part-time / full-time)
Oct 7 NEED HANDYMAN FOR TASKS - at least $35/hr, Paid Daily
Oct 7 Warehouse Team Member - $3,000 Sign-On Bonus
Oct 7 Warehouse Team Member - $3,000 Sign-On Bonus
Oct 7 Amazon Warehouse Assistant - Immediate Hire
Oct 7 Amazon Picker - Overnight Shifts Available
Oct 7 Warehouse Team Member - With Sign-On Bonus!
Oct 7 Amazon Warehouse Assistant
Oct 7 Need a Crew of 4+
Oct 6 Helper and Construction
Oct 6 Truck Driver/El Camionero
Oct 6 Workers Needed for Local Fair (Cash Paid Daily)
Oct 6 Roofing Crews Needed! ($2,000 Sign On Bonus - Start Today!)
Oct 6 Available Immediately General Labor/Driver with Truck or Van Positions
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 Automotive Technicians Needed
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 Automotive Technicians Needed
Oct 6 Automotive Technicians Needed
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 NEED HANDYMAN FOR TASKS - at least $35/hr, Paid Daily
Oct 6 $20-25/hr. SIMPLE VALET TRASH COLLECTOR-Camp Creek/East Point 30331
Oct 6 Asphalt workers labors
Oct 6 CDL-A Company Drivers - $5K SIGN-ON BONUS! 💰 🚚
Oct 6 EXPERIENCED HANDYMAN ***2 immediate openings
Oct 6 Automotive Detail /Porter/Shop Help
Oct 6 Fork Lift Operator/ Order Picker
Oct 6 Lot Operations
Oct 6 Waterproofing Foreman
Oct 6 Auto Detailer
Oct 6 Lyft Driver (part-time / full-time)
Oct 6 Lyft Driver (part-time / full-time)
Oct 6 FULL TIME WITH PAID BENEFITS $16+ HOUR

>Solution :

Try:

import requests
from datetime import datetime
from bs4 import BeautifulSoup

link = "https://atlanta.craigslist.org/search/lab"

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36",
}

days_to_go_back = 2

with requests.Session() as s:
    s.headers.update(headers)
    res = s.get(link)
    soup = BeautifulSoup(res.text, "lxml")

    now = datetime.now()

    for item in soup.select("[id^='search-results'] li[class*='result']"):
        date = " ".join(item.select_one("time.result-date").text.split())
        title = item.select_one("h3.result-heading > a").get_text(strip=True)

        d = datetime.strptime(item.time["datetime"][:10], "%Y-%m-%d")
        if (now - d).days < days_to_go_back:
            print(date, title)

Prints:


...
Oct 6 Automotive Detail /Porter/Shop Help
Oct 6 Fork Lift Operator/ Order Picker
Oct 6 Lot Operations
Oct 6 Waterproofing Foreman
Oct 6 Auto Detailer
Oct 6 Lyft Driver (part-time / full-time)
Oct 6 Lyft Driver (part-time / full-time)
Oct 6 FULL TIME WITH PAID BENEFITS $16+ HOUR
byMR
Published
Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Read more

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading